2

Right now I have events being forwarded to a collector on which I then run a script to pull them out of the collector's event logs as xml and then insert them in a DB.

Is there any utility that knows the native windows event forwarding protocol that can receive incoming forwarded events and insert them directly into a DB? I'm surprised I haven't seen any products that do this yet.

I was looking at the API, but it doesn't look like it provides a way to change the data source to something like a sql DB instead of the windows eventlog.

red888
  • 4,253

3 Answers3

1

Use LogParser to extract data from Windows Event logs (among many sources) and replicate them into a database. If run on a schedule, it can update the database with any new events.

Pekka
  • 530
  • hmmm would the performance of this be superior to wevtutil.exe do you think? That's what I'm using now, but I'll check out LogParser. – red888 Mar 12 '14 at 14:37
  • Not likely, but it allows super simple, flexible, configuration with schedule retrieval. It is not really intended for alert polling, but can be useful for mashing up and testing concepts quickly. – Pekka Mar 15 '14 at 09:31
1

The performance of such a design would not scale up well. It may work in small/medium environments though.

The Windows Event Collector is designed to do just that - scale up on a massive level. A single collector server can collect from hundreds or thousands of computers, 10's of GB or even 100's GB per day.

On comparable hardware, a pull subscriber approach on a SQL server would not come close to matching the performance of the Windows Event Collector. And a push subscriber approach on a SQL server - even with minimal indexes would be flattened like wile coyote long before reaching anything that a Windows Event Collector could do.

And you would not want to compete with the inserts, so in large environments, a reporting database for queries may be necessary, especially if you have a lot of related tables or carpet-bomb them with indexes. So at scale the solution would not be able to compete on either performance or cost.

There is a significant impedance mismatch in performance between the Event Collector and SQL server. That's why the Event Collector is such a good fit as a transformation intermediary. The transformation apps/scripts that read the Forwarded Events log are inexpensive and easy to build, so there is no business justification (green stuff) for eliminating them. The SQL server would only need a single database - no need for a reporting database, and could be of modest capacity, because the transformation could be performed in large batches with minimal impact on the database server.

Greg Askew
  • 36,724
  • What you say makes a lot of sense. Frankly I just didn't want to have to fix my script which currently uses wevtutil to return a MASSIVE amount of events every 2 hours and then does a big ass insert with Data.SqlClient.SqlBulkCopy- its pretty dumb. Ultimately I'll try LogParser and actually do some pagination. – red888 Mar 12 '14 at 14:44
0

You could do that with syslog-ng, which has client software for Windows machines.

A syslog-ng server can store its logs in a database, both relational and non-relational.

dawud
  • 15,286
  • 1
    Well that's exactly what I'm trying to avoid. I using windows native event forwarding which doesn't require client software. – red888 Mar 11 '14 at 20:55