-bash-4.1$ cat crash-2018-05-22-13:02:27.log
(Out of file descriptors!)
[build 2e75b3406c5b] 2018-05-22 13:02:27
This is the error that I am seeing
Search peer (server_name) has the following message: Bundle Replication: Problem replicating config (bundle) to search peer ' (Server IP):8089 ', HTTP response code 409 (HTTP/1.1 409 Conflict). Failed to untar the bundle="/opt/splunk/var/run/searchpeers/(server_name)-1527015758.bundle". This could be due Search Head attempting to upload the same bundle again after a timeout. Check for sendRcvTimeout message in splund.log, consider increasing it.
And this is the crash log:
File descriptors open:
0: /opt/splunk/var/log/splunk/crash-2018-05-22-13:02:27.log
1: /opt/splunk/var/log/splunk/splunkd_stdout.log
2: /opt/splunk/var/log/splunk/splunkd_stderr.log
3: /opt/splunk/var/log/splunk/splunkd.log
4: socket:[27180755]
5: socket:[27180756]
6: socket:[27180757]
7: socket:[27180761]
8: [eventpoll]
9: socket:[27180763]
10: [eventfd]
11: pipe:[27180765]
12: /opt/splunk/var/log/splunk/audit.log
13: /opt/splunk/var/log/splunk/license_usage.log
14: [eventfd]
15: [eventpoll]
16: /opt/splunk/share/splunk/mbtiles/splunk-tiles.mbtiles
17: [eventfd]
18: socket:[27191160]
19: [eventpoll]
20: [eventpoll]
21: [eventfd]
22: [eventpoll]
23: [eventfd]
24: /opt/splunk/var/log/splunk/conf.log
25: [eventfd]
26: [eventpoll]
27: [eventfd]
28: [eventpoll]
29: [eventfd]
30: pipe:[27180848]
31: [eventpoll]
32: [eventfd]
33: /opt/splunk/var/log/splunk/mongod.log
34: /opt/splunk/var/lib/splunk/fishbucket/splunk_private_db/btree_index.dat
35: [eventfd]
36: socket:[27180852]
37: [eventpoll]
38: /opt/splunk/var/lib/splunk/fishbucket/splunk_private_db/btree_records.dat
39: [eventfd]
40: [eventfd]
41: [eventpoll]
42: [eventfd]
43: [eventpoll]
44: [eventfd]
45: [eventfd]
46: [eventpoll]
47: [eventpoll]
48: [eventpoll]
49: [eventfd]
50: [eventfd]
51: [eventpoll]
52: [eventpoll]
53: [eventfd]
54: [eventpoll]
55: [eventfd]
56: pipe:[27180877]
57: socket:[27196966]
58: pipe:[27180878]
59: socket:[27183669]
60: socket:[27180902]
61: /opt/splunk/var/log/splunk/metrics.log
62: /opt/splunk/var/lib/splunk/persistentstorage/fschangemanager_state
63: [eventpoll]
64: [eventfd]
65: [eventpoll]
66: [eventfd]
67: [eventpoll]
68: [eventfd]
69: [eventpoll]
70: [eventfd]
71: socket:[27191201]
72: [eventpoll]
73: [eventpoll]
74: socket:[27197043]
75: pipe:[27181879]
76: pipe:[27181910]
77: /opt/splunk/share/splunk/mbtiles/splunk-tiles.mbtiles
78: socket:[27183769]
79: socket:[27183773]
80: socket:[27183625]
81: socket:[27210557]
82: [eventfd]
83: [eventfd]
84: [eventpoll]
85: [eventfd]
86: /opt/splunk/var/log/introspection/disk_objects.log
87: [eventfd]
88: [eventpoll]
89: pipe:[27181021]
90: [eventfd]
91: [eventpoll]
92: /opt/splunk/var/log/splunk/health.log
93: /opt/splunk/var/log/splunk/splunkd_access.log
94: [eventpoll]
95: pipe:[27181880]
96: [eventpoll]
97: socket:[27205883]
98: [eventfd]
99: pipe:[27181911]
100: /opt/splunk/var/log/introspection/http_event_collector_metrics.log
101: socket:[27187915]
102: socket:[27183631]
103: [eventfd]
104: socket:[27183627]
105: socket:[27191162]
106: /opt/splunk/var/log/splunk/splunkd_ui_access.log
107: [eventfd]
108: [eventpoll]
109: socket:[27191244]
110: socket:[27212042]
111: [eventfd]
112: socket:[27183629]
113: [eventpoll]
114: socket:[27200480]
115: socket:[27216864]
116: /opt/splunk/var/log/introspection/kvstore.log
117: /opt/splunk/var/log/splunk/splunkd_ui_access.log
118: socket:[27191203]
119: /opt/splunk/var/log/introspection/resource_usage.log
120: /opt/splunk/var/log/splunk/scheduler.log
121: pipe:[27183400]
122: pipe:[27183400]
123: [eventpoll]
124: [eventfd]
125: /opt/splunk/var/log/splunk/remote_searches.log
126: socket:[27191246]
127: [eventpoll]
128: socket:[27204214]
129: socket:[27187283]
130: socket:[27196968]
131: socket:[27200482]
132: socket:[27187917]
133: socket:[27189987]
134: socket:[27189988]
135: socket:[27189989]
136: socket:[27189990]
137: /opt/splunk/var/log/splunk/splunkd_access.log
138: /etc/cma.d/lpc.conf
139: [eventfd]
140: socket:[27216866]
141: [eventpoll]
142: socket:[27204385]
143: socket:[27204216]
144: socket:[27222271]
145: /opt/splunk/var/log/splunk/audit.log
146: socket:[27222191]
147: socket:[27217115]
148: socket:[27197045]
149: socket:[27211889]
150: /opt/splunk/var/log/splunk/scheduler.log
151: socket:[27204387]
153: socket:[27222273]
157: socket:[27205754]
158: socket:[27205972]
159: socket:[27205756]
160: socket:[27217117]
163: socket:[27205448]
166: socket:[27205510]
167: socket:[27205450]
169: socket:[27205555]
170: socket:[27205512]
171: socket:[27205618]
172: socket:[27205557]
173: socket:[27205885]
174: socket:[27205620]
175: socket:[27206050]
176: socket:[27205974]
178: socket:[27206027]
179: socket:[27206112]
180: socket:[27206052]
181: socket:[27210559]
182: socket:[27206114]
188: socket:[27211891]
189: socket:[27212044]
(Total 175)
Received fatal signal 6 (Aborted).
Cause:
Signal sent by PID 10152 running under UID 18002.
Crashing thread: GenerationGrabberThread
Registers:
RIP: [0x00007F8BF2848495] gsignal + 53 (libc.so.6 + 0x32495)
RDI: [0x00000000000027A8]
RSI: [0x0000000000004390]
RBP: [0x00007F8BF5F21D80]
RSP: [0x00007F8BD37FD538]
RAX: [0x0000000000000000]
RBX: [0x00007F8BF3DE0000]
RCX: [0xFFFFFFFFFFFFFFFF]
RDX: [0x0000000000000006]
R8: [0x00000000000003F8]
R9: [0xFEFEFEFEFEFEFEFF]
R10: [0x0000000000000008]
R11: [0x0000000000000202]
R12: [0x00007F8BF5E5DFAF]
R13: [0x00007F8BF6013600]
R14: [0x00007F8BD37FDDA0]
R15: [0x00007F8BD37FD840]
EFL: [0x0000000000000202]
TRAPNO: [0x0000000000000000]
ERR: [0x0000000000000000]
CSGSFS: [0x0000000000000033]
OLDMASK: [0x0000000000000000]
OS: Linux
Arch: x86-64
Backtrace (PIC build):
Linux / (servernamehere) / 2.6.32-696.28.1.el6.x86_64 / #1 SMP Thu Apr 26 04:27:41 EDT 2018 / x86_64
glibc version: 2.12
glibc release: stable
Last errno: 23
Threads running: 76
Runtime: 471.198094s
argv: [splunkd -p 8089 start]
Regex JIT disabled due to SELinux
using CLOCK_MONOTONIC
Thread: "GenerationGrabberThread", did_join=0, ready_to_run=Y, main_thread=N
First 8 bytes of Thread token @0x7f8be9613090:
00000000 00 f7 7f d3 8b 7f 00 00 |........|
00000008
x86 CPUID registers:
0: 0000000D 756E6547 6C65746E 49656E69
1: 000206D2 09040800 9E982203 1F8BFBFF
2: 76036301 00F0B5FF 00000000 00C10000
3: 00000000 00000000 00000000 00000000
4: 00000000 00000000 00000000 00000000
5: 00000000 00000000 00000000 00000000
6: 00000077 00000002 00000009 00000000
7: 00000000 00000000 00000000 00000000
8: 00000000 00000000 00000000 00000000
9: 00000000 00000000 00000000 00000000
A: 07300401 0000007F 00000000 00000000
B: 00000000 00000000 000000CD 00000009
C: 00000000 00000000 00000000 00000000
D: 00000000 00000000 00000000 00000000
80000000: 80000008 00000000 00000000 00000000
80000001: 00000000 00000000 00000001 28100800
80000002: 65746E49 2952286C 6F655820 2952286E
80000003: 55504320 2D354520 37383632 33762057
80000004: 33204020 4730312E 00007A48 00000000
80000005: 00000000 00000000 00000000 00000000
80000006: 00000000 00000000 01006040 00000000
80000007: 00000000 00000000 00000000 00000100
80000008: 00003028 00000000 00000000 00000000
terminating...
-bash-4.1$
↧