[.kibana_task_manager] Action failed with ‘search_phase_execution_exception

发布于 2024-11-08  52.78k 次阅读


ES版本:7.17.5

Kinaba版本:7.17.5

kibana_task_manager 索引损坏

  • 如果 kibana_task_manager 索引中的数据损坏或出现异常,Kibana 启动时会无法加载该索引,导致启动失败。

{"type":"log","@timestamp":"2024-11-08T14:03:58+08:00","tags":["error","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] Action failed with 'search_phase_execution_exception: '. Retrying attempt 12 in 64 seconds."}
{"type":"log","@timestamp":"2024-11-08T14:03:58+08:00","tags":["info","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 64016ms."}
{"type":"log","@timestamp":"2024-11-08T14:05:02+08:00","tags":["error","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] Action failed with 'search_phase_execution_exception: '. Retrying attempt 13 in 64 seconds."}
{"type":"log","@timestamp":"2024-11-08T14:05:02+08:00","tags":["info","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 64020ms."}
{"type":"log","@timestamp":"2024-11-08T14:06:06+08:00","tags":["error","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] Action failed with 'search_phase_execution_exception: '. Retrying attempt 14 in 64 seconds."}
{"type":"log","@timestamp":"2024-11-08T14:06:06+08:00","tags":["info","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 64016ms."}
{"type":"log","@timestamp":"2024-11-08T14:07:10+08:00","tags":["error","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] Action failed with 'search_phase_execution_exception: '. Retrying attempt 15 in 64 seconds."}
{"type":"log","@timestamp":"2024-11-08T14:07:10+08:00","tags":["info","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 64014ms."}
{"type":"log","@timestamp":"2024-11-08T14:08:14+08:00","tags":["info","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> FATAL. took: 64017ms."}
{"type":"log","@timestamp":"2024-11-08T14:08:14+08:00","tags":["fatal","root"],"pid":7,"message":"Error: Unable to complete saved object migrations for the [.kibana_task_manager] index: Unable to complete the OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT step after 15 attempts, terminating.\n    at migrationStateActionMachine (/usr/share/kibana/src/core/server/saved_objects/migrationsv2/migrations_state_action_machine.js:144:29)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at async Promise.all (index 1)\n    at SavedObjectsService.start (/usr/share/kibana/src/core/server/saved_objects/saved_objects_service.js:181:9)\n    at Server.start (/usr/share/kibana/src/core/server/server.js:330:31)\n    at Root.start (/usr/share/kibana/src/core/server/root/index.js:69:14)\n    at bootstrap (/usr/share/kibana/src/core/server/bootstrap.js:120:5)\n    at Command.<anonymous> (/usr/share/kibana/src/cli/serve/serve.js:229:5)"}
{"type":"log","@timestamp":"2024-11-08T14:08:14+08:00","tags":["info","plugins-system","standard"],"pid":7,"message":"Stopping all plugins."}
{"type":"log","@timestamp":"2024-11-08T14:08:14+08:00","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":7,"message":"Monitoring stats collection is stopped"}
{"type":"log","@timestamp":"2024-11-08T14:08:44+08:00","tags":["warning","plugins-system","standard"],"pid":7,"message":"\"eventLog\" plugin didn't stop in 30sec., move on to the next."}

 FATAL  Error: Unable to complete saved object migrations for the [.kibana_task_manager] index: Unable to complete the OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT step after 15 attempts, terminating.

[root@localhost data]# curl http://localhost:9200/_cat/aliases | grep kibana
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3258  100  3258    0     0   102k      0 --:--:-- --:--:-- --:--:--  102k
.kibana                      .kibana_7.17.5_001              - - - -
.kibana_7.17.5               .kibana_7.17.5_001              - - - -
.kibana-event-log-7.17.5     .kibana-event-log-7.17.5-000003 - - - false
.kibana_task_manager         .kibana_task_manager_7.17.5_001 - - - -
.kibana_task_manager_7.17.5  .kibana_task_manager_7.17.5_001 - - - -
.kibana-event-log-7.17.5     .kibana-event-log-7.17.5-000002 - - - false
.kibana-event-log-7.17.5     .kibana-event-log-7.17.5-000004 - - - true
.kibana-event-log-7.17.5     .kibana-event-log-7.17.5-000001 - - - false

[root@localhost data]# curl -X DELETE http://localhost:9200/.kibana_task_manager_7.17.5_001
{"acknowledged":true}
[root@localhost data]# curl -X DELETE http://localhost:9200/.kibana_7.17.5_001
{"acknowledged":true}

删除索引重建即可