Introduction of Appflow support and JSON data-output support for collector (templates still to DB) - json_milestone_1 #1
Conversation
Link in new assets into build. Fix up missing dependency in Makefile.
get mapped as something like 5951_129 while others are fine. Also helps if you include the actual definitions.
for building the IE data (description has comments). Various IEs were not getting created for Netscaler. It now uses the pipe '|'. Looking good. Just need a more up-to-date IE description, as there are IEs seen that are above 269 (the highest currently known).
Symptom was that strmov from -lmysqlclient was giving a segfault immediately on call; even though ti was my_inet_ntoa that was called.
trying to create a table with duplicate columns. Similarly, don't try and populate such a column.
…_ident() failed"
…be NULL if not used). But it doesn't do anything yet.
Note that 64-bit will need to change to a string representation perhaps. IP (4&6) addresses and NTP timestamps are the largest omissions yet.
Support for pkg-config
REFACTOR REQUIRED Needed to refactor the codebase slightly to make the necessary global datastructures visible to the relevant code.
cameronkerrnz
changed the title
This undoes some previous work, and also fixes up some minor autoconf issues for RHEL6-based builders. Resolves #20 Completely divorce the JSON emitter from MySQL, making a separate collector
empty in this case means "" for strings, or a 0-byte array for bytes.
ipfix_init always uses the current time for the start time, which is not ideal and means that we can't control the exported uptime. The ipfix_init_with_start_time function takes a time_t to calculate uptime from. Signed-off-by: James Wheatley <[email protected]>
|
Hi, have just started looking at using this in a similar context to your own, however I'm finding that a lot of the events are not being output correctly. I'm only looking at inbound HTTP events, and finding that over 30% of the events aren't getting data into the right fields - such as getting the URI in netscaler_http_req_method, the referrer in netscaler_http_req_user_agent, and the UA in netscaler_http_req_via. Some of the data appears to have crossed field boundaries and includes unprintable characters from the packet headers in the text. Have you experienced anything similar to this? I'm afraid I will be limited in my ability to provide examples but I will help as much as I can. My main stumbling block is that I don't really know C so figuring out where it's breaking is a bit beyond me. |
|
Hi. I can't give any help without some samples of what has gone wrong. Anonymise if you like. Are you using a Netscaler? If so, which version? A packet capture and dump of the database containing what schemes have been obtained would be useful. It's been some time since I last worked on this... so pretty rusty. Sent from my iPhone
|
|
Hi, I really appreciate that you responded, and so promptly! I'm afraid that a packet capture isn't something I would be able to provide as it would be too hard to anonymise, however I believe it would be feasible to provide a sample of the broken JSON. As for the DB dump, please could you say where it would be located and what format? It may be possible to anonymise, but not sure. Yes it is a Netscaler, I can send over the version with the data I am able to provide in an email if that's ok? |
Added ipfix_init_with_start_time function
As my use-case is around getting data into ELK, I'm migrating my efforts to https://github.com/logstash-plugins/logstash-codec-netflow. All development on this fork of libipfix will hereby cease.
Hello, I've been working on libipfix for an integration project I've been working on, getting logs from a Citrix Netscaler appliance (ie. Appflow, which is a particular set of template extensions for IPFIX), and forwarding integrating this data with the ELK framework (ElasticSearch, Logstash and Kibana), in which case its much preferable for me to send the logs as JSON.
I have also fixed a couple minor bugs. I haven't bumped the version at this time, but I have a milestone tag called 'json_milestone_1' which relates to this Pull Request.
I won't say yet that this is ready for production use; there are a couple more milestones I want to accomplish before I consider it ready for use in production, but it should be usable to at least receive real-world data and do something generally useful with it.
As such, you may like to see this Pull Request merely as a means to let you know what I've been doing. Feel free to have a look at my Issues list and milestones to see what I'm planning.
Thanks for your work on this; its been a very useful base on which to work on.
Cheers,
Cameron