모든 쿼리를 기록하는 MongoDB

모든 쿼리를 기록하는 MongoDB

---

질문은 간단합니다. 기본적으로 mongodb의 "꼬리"가능한 로그 파일에 모든 쿼리를 어떻게 기록합니까?

나는 시도했다 :

• 프로파일 링 레벨 설정
• 느린 ms 매개 변수 시작 설정
• -vv 옵션을 가진 mongod

/var/log/mongodb/mongodb.log는 현재 활성 연결 수만 계속 표시합니다 ...

---

모든 쿼리를 기록 할 수 있습니다.

$ mongo
MongoDB shell version: 2.4.9
connecting to: test
> use myDb
switched to db myDb
> db.getProfilingLevel()
0
> db.setProfilingLevel(2)
{ "was" : 0, "slowms" : 1, "ok" : 1 }
> db.getProfilingLevel()
2
> db.system.profile.find().pretty()

출처 : http://docs.mongodb.org/manual/reference/method/db.setProfilingLevel/

db.setProfilingLevel(2) "모든 작업 기록"을 의미합니다.

---

나는 이런 식으로 mongod를 시작하여 이것을 해결했다. (망치고 못생긴, 그래 ...하지만 개발 환경에서 작동한다) :

mongod --profile=1 --slowms=1 &

그러면 프로파일 링이 가능하고 "느린 쿼리"에 대한 임계 값을 1ms로 설정하여 모든 쿼리가 파일에 "느린 쿼리"로 기록됩니다.

/var/log/mongodb/mongodb.log

이제 다음 명령을 사용하여 지속적인 로그 출력을 얻습니다.

tail -f /var/log/mongodb/mongodb.log

로그 예 :

Mon Mar  4 15:02:55 [conn1] query dendro.quads query: { graph: "u:http://example.org/people" } ntoreturn:0 ntoskip:0 nscanned:6 keyUpdates:0 locks(micros) r:73163 nreturned:6 reslen:9884 88ms

---

구글 최초의 답변이기 때문에 ...
버전 3

$ mongo
MongoDB shell version: 3.0.2
connecting to: test
> use myDb
switched to db
> db.setLogLevel(1)

http://docs.mongodb.org/manual/reference/method/db.setLogLevel/

---

MongoDB정교한 프로파일 링 기능이 있습니다. 로깅은 system.profile수집 에서 발생합니다 . 로그는 다음에서 볼 수 있습니다.

db.system.profile.find()

3 가지 로깅 레벨 ( source )이 있습니다.

• 레벨 0- 프로파일 러가 꺼져 있고 데이터를 수집하지 않습니다. mongod는 항상 slowOpThresholdMs 임계 값보다 긴 작업을 로그에 씁니다. 이것이 기본 프로파일 러 수준입니다.
• 레벨 1- 느린 조작에 대해서만 프로파일 링 데이터를 수집합니다. 기본적으로 느린 작업은 100 밀리 초보다 느린 작업입니다. slowOpThresholdMs 런타임 옵션 또는 setParameter 명령을 사용하여 "느린"작업에 대한 임계 값을 수정할 수 있습니다. 자세한 내용은 느린 작업에 대한 임계 값 지정 섹션을 참조하십시오.
• 레벨 2- 모든 데이터베이스 조작에 대한 프로파일 링 데이터를 수집합니다.

데이터베이스가 실행중인 프로파일 링 레벨을 확인하려면 다음을 사용하십시오.

db.getProfilingLevel()

그리고 상태를 볼 수

db.getProfilingStatus()

프로파일 링 상태를 변경하려면 다음 명령을 사용하십시오.

db.setProfilingLevel(level, milliseconds)

여기서 level프로파일 링 수준 milliseconds을 나타내며 쿼리를 기록해야하는 기간 (ms)입니다. 로깅을 끄려면

db.setProfilingLevel(0)

The query to look in the system profile collection for all queries that took longer than one second, ordered by timestamp descending will be

db.system.profile.find( { millis : { $gt:1000 } } ).sort( { ts : -1 } )

---

I made a command line tool to activate the profiler activity and see the logs in a "tail"able way: "mongotail".

But the more interesting feature (also like tail) is to see the changes in "real time" with the -f option, and occasionally filter the result with grep to find a particular operation.

See documentation and installation instructions in: https://github.com/mrsarm/mongotail

---

Once profiling level is set using db.setProfilingLevel(2).

The below command will print the last executed query.
You may change the limit(5) as well to see less/more queries.
$nin - will filter out profile and indexes queries
Also, use the query projection {'query':1} for only viewing query field

db.system.profile.find(
{ 
    ns: { 
        $nin : ['meteor.system.profile','meteor.system.indexes']
    }
} 
).limit(5).sort( { ts : -1 } ).pretty()

Logs with only query projection

db.system.profile.find(
{ 
    ns: { 
        $nin : ['meteor.system.profile','meteor.system.indexes']
    }
},
{'query':1}
).limit(5).sort( { ts : -1 } ).pretty()

---

The profiler data is written to a collection in your DB, not to file. See http://docs.mongodb.org/manual/tutorial/manage-the-database-profiler/

I would recommend using 10gen's MMS service, and feed development profiler data there, where you can filter and sort it in the UI.

---

if you want the queries to be logged to mongodb log file, you have to set both the log level and the profiling, like for example:

db.setLogLevel(1)
db.setProfilingLevel(2)

(see https://docs.mongodb.com/manual/reference/method/db.setLogLevel)

Setting only the profiling would not have the queries logged to file, so you can only get it from

db.system.profile.find().pretty()

---

I think that while not elegant, the oplog could be partially used for this purpose: it logs all the writes - but not the reads...

You have to enable replicatoon, if I'm right. The information is from this answer from this question: How to listen for changes to a MongoDB collection?

---

Setting profilinglevel to 2 is another option to log all queries.

---

I recommend checking out mongosniff. This can tool can do everything you want and more. Especially it can help diagnose issues with larger scale mongo systems and how queries are being routed and where they are coming from since it works by listening to your network interface for all mongo related communications.

http://docs.mongodb.org/v2.2/reference/mongosniff/

---

I wrote a script that will print out the system.profile log in real time as queries come in. You need to enable logging first as stated in other answers. I needed this because I'm using Windows Subsystem for Linux, for which tail still doesn't work.

https://github.com/dtruel/mongo-live-logger

---

db.adminCommand( { getLog: "*" } )

Then

db.adminCommand( { getLog : "global" } )

---

Try out this package to tail all the queries (without oplog operations): https://www.npmjs.com/package/mongo-tail-queries

(Disclaimer: I wrote this package exactly for this need)

---

This was asked a long time ago but this may still help someone:

MongoDB profiler logs all the queries in the capped collection system.profile. See this: database profiler

1. Start mongod instance with --profile=2 option that enables logging all queries OR if mongod instances is already running, from mongoshell, run db.setProfilingLevel(2) after selecting database. (it can be verified by db.getProfilingLevel(), which should return 2)
2. After this, I have created a script which utilises mongodb's tailable cursor to tail this system.profile collection and write the entries in a file. To view the logs I just need to tail it:tail -f ../logs/mongologs.txt. This script can be started in background and it will log all the operation on the db in the file.

My code for tailable cursor for the system.profile collection is in nodejs; it logs all the operations along with queries happening in every collection of MyDb:

const MongoClient = require('mongodb').MongoClient;
const assert = require('assert');
const fs = require('fs');
const file = '../logs/mongologs'
// Connection URL
const url = 'mongodb://localhost:27017';

// Database Name
const dbName = 'MyDb';
//Mongodb connection

MongoClient.connect(url, function (err, client) {
   assert.equal(null, err);
   const db = client.db(dbName);
   listen(db, {})
});

function listen(db, conditions) {
var filter = { ns: { $ne: 'MyDb.system.profile' } }; //filter for query
//e.g. if we need to log only insert queries, use {op:'insert'}
//e.g. if we need to log operation on only 'MyCollection' collection, use {ns: 'MyDb.MyCollection'}
//we can give a lot of filters, print and check the 'document' variable below

// set MongoDB cursor options
var cursorOptions = {
    tailable: true,
    awaitdata: true,
    numberOfRetries: -1
};

// create stream and listen
var stream = db.collection('system.profile').find(filter, cursorOptions).stream();

// call the callback
stream.on('data', function (document) {
    //this will run on every operation/query done on our database
    //print 'document' to check the keys based on which we can filter
    //delete data which we dont need in our log file

    delete document.execStats;
    delete document.keysExamined;
    //-----
    //-----

    //append the log generated in our log file which can be tailed from command line
    fs.appendFile(file, JSON.stringify(document) + '\n', function (err) {
        if (err) (console.log('err'))
    })

});

}

For tailable cursor in python using pymongo, refer the following code which filters for MyCollection and only insert operation:

import pymongo
import time
client = pymongo.MongoClient()
oplog = client.MyDb.system.profile
first = oplog.find().sort('$natural', pymongo.ASCENDING).limit(-1).next()

ts = first['ts']
while True:
    cursor = oplog.find({'ts': {'$gt': ts}, 'ns': 'MyDb.MyCollection', 'op': 'insert'},
                        cursor_type=pymongo.CursorType.TAILABLE_AWAIT)
    while cursor.alive:
        for doc in cursor:
            ts = doc['ts']
            print(doc)
            print('\n')
        time.sleep(1)

Note: Tailable cursor only works with capped collections. It cannot be used to log operations on a collection directly, instead use filter: 'ns': 'MyDb.MyCollection'

Note: I understand that the above nodejs and python code may not be of much help for some. I have just provided the codes for reference.

Use this link to find documentation for tailable cursor in your languarge/driver choice Mongodb Drivers

Another feature that i have added after this logrotate.

참고URL : https://stackoverflow.com/questions/15204341/mongodb-logging-all-queries