Host is not visible via API


(Igor Petrov) #1

Hello,

my initial question is on old forum https://monitoring-portal.org/woltlab/index.php?thread/42633-host-is-not-visible-via-api/#post260174

I upgraded to r2.8.1-1, but see exactly the same problem again.
The host was added via API after upgrade, so this is not an old data.
The problem is intermittent: it reproduces only for some checks.

Was it fixed in 2.8 ? Is there a workaround? Maybe I need to clean cache?
I tried to drop /var/cache/icinga2/* but it did not help.

Thank You


#2

Could you post the curl output here again? With the output of the offending file if possible, a icinga2 object list --name $NameOfTheProblematicHost
You will need to delete the offending file to recreate the Host.


(Igor Petrov) #3

CURL output:

curl -k -s -u root:icinga2apipassword -X GET -H 'Accept: application/json' 'https://localhost:5665/v1/objects/hosts'
{"results":[]}

curl --insecure --user root:icinga2apipassword --request "PUT" --header "Accept: application/json" --data curl --insecure --user root:icinga2apipassword --request "PUT" --header "Accept: application/json" --data '{"templates":["generic-host"],"attrs":{"address":"www.example.com", "check_command":"dummy","zone":"MyCompany"}}' https://localhost:5665/v1/objects/hosts/MyCompany_dummy_example
{"results":[{"code":500.0,"errors":["Configuration file '/var/lib/icinga2/api/packages/_api//conf.d/hosts/MyCompany_dummy_example.conf' already exists."],"status":"Object could not be created."}]}

icinga2 object list --name MyCompany_dummy_example
# empty output

# icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.8.1-1)

Host MyCompany_dummy_example was created via API on a clean Icinga2 instance.

Any ideas? Is it Icinga2 bug?


(Michael Friedrich) #4

Sounds like the _api package is broken, there is the stage name missing between _api/ and /conf.d. Did you modify something in there by yourself?


(Igor Petrov) #5

No, the host was created via API PUT request. I did not edit there anything manually.


(Michael Friedrich) #6

Sounds like https://github.com/Icinga/icinga2/issues/3668#issuecomment-282549005


(Igor Petrov) #7

Shouldn’t it be fixed in 2.8 which we are using?
We are constantly hitting this bug. Now on a satellite in our cluster I see configs replicated from master:

satellite# ls -la /var/lib/icinga2/api/packages/_api/conf.d/services/
host1!disk.conf
host1!proc.conf

There is no stage name between /api/ and /conf.d/ (we did not do anything manually on satellite, all config distribution goes via master)
Those checks are in PENDING state - satellite does not pick them up (GET /v1/objects/services/ does not show them).

Shall I open a new bug?
If you confirm that this is a bug, we can try to contribute to the project to fix it. Could you please give us some directions in this area?


(Igor Petrov) #8

When I restart our satellite, I see errors:

[2018-05-18 15:09:34 +0000] information/ApiListener: New client connection for identity 'icinga2' to [0.0.0.0]:5665
[2018-05-18 15:09:34 +0000] information/ApiListener: Finished reconnecting to endpoint 'icinga2' via host 'masterhost' and port '5665'
[2018-05-18 15:09:34 +0000] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint 'icinga2'.
[2018-05-18 15:09:34 +0000] information/ApiListener: Sending config updates for endpoint 'icinga2' in zone 'MyCompany'.
[2018-05-18 15:09:34 +0000] information/ApiListener: Finished sending config file updates for endpoint 'icinga2' in zone 'MyCompany'.
[2018-05-18 15:09:34 +0000] information/ApiListener: Syncing runtime objects to endpoint 'icinga2'.
[2018-05-18 15:09:34 +0000] information/ApiListener: Finished syncing runtime objects to endpoint 'icinga2'.
[2018-05-18 15:09:34 +0000] information/ApiListener: Finished sending runtime config updates for endpoint 'icinga2' in zone 'MyCompany'.
[2018-05-18 15:09:34 +0000] information/ApiListener: Sending replay log for endpoint 'icinga2' in zone 'MyCompany'.
[2018-05-18 15:09:34 +0000] information/ApiListener: Finished sending replay log for endpoint 'icinga2' in zone 'MyCompany'.
[2018-05-18 15:09:34 +0000] information/ApiListener: Finished syncing endpoint 'icinga2' in zone 'MyCompany'.
[2018-05-18 15:09:34 +0000] information/ApiListener: Applying config update from endpoint 'icinga2' of zone 'MyCompany'.
[2018-05-18 15:09:35 +0000] critical/ApiListener: Could not create object 'host1':
[2018-05-18 15:09:35 +0000] critical/ApiListener: Configuration file '/var/lib/icinga2/api/packages/_api//conf.d/hosts/host1.conf' already exists.

icinga2 is master endpoint.
We are running r2.8.3-1 and we did not modify anything on satellite manually, all config updates come from master.
I can clearly see that stage is missing.

Any ideas?


#9

there are 3 ways to create host objects:
a) static config files and their replication via the masters zones.d directory
b) single object via the api
c) multiple objects in packages, via the api (that is what director does).

So, it looks for me that in the named config package file an host object “host1” already exists but config replication tries to recreate it.


(Igor Petrov) #10

Thank you for your reply.
Yes, that is correct. Bu why is it happening? Is it a bug?
The scenario to reproduce it is:

  1. setup a cluster
  2. deploy two host objects via API (not via director)
  3. restart satellite

The path about which Icinga complains ('/var/lib/icinga2/api/packages/_api//conf.d/hosts/host1.conf') looks wrong because of /_api//conf.d/ - like some variable is empty.

Are we doing something wrong in our setup?


#11

Sorry, at the time i wrote my reply the thread started with what has number 7 by now -
I was living in the assumption that there have not been any replies to your question.
Not sure if i made a mistake or if my personal thread view went nuts.


(Igor Petrov) #12

Thank You.

So, can anybody comment on this - is it a bug or we are doing something wrong?


(Michael Friedrich) #13

Attempt to fix it like described here and watch if it is reproducibly failing again.


(Igor Petrov) #14

We tried. And it is failing again.
We can’t apply this workaround every time.
Shall I open a bug?