Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 504872 reports (Page 5555 of 10098)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [......X........] duration: 56.648166ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/1ed0d71156ecf368 FAIL (2025-10-31 22:17:10 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:49584->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 56.787592ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............................X........................X...] duration: 3.606316674s trace: https://jaeger.dev.storj.io/trace/150d6d8439e49137 FAIL (2025-10-31 22:17:02 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.606345789s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 8.110581423s trace: https://jaeger.dev.storj.io/trace/4a436e8f0e3652a9 FAIL (2025-10-31 22:16:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== =QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841512922 (34h13m43.537361665s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841695811 (34h13m43.537544554s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842121341 (34h13m43.537970084s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842258863 (34h13m43.538107606s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842653173 (34h13m43.538501917s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843123740 (34h13m43.538972493s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843233278 (34h13m43.539082031s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843685530 (34h13m43.539534283s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844090440 (34h13m43.539939193s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844193336 (34h13m43.540042089s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844586345 (34h13m43.540435098s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844901665 (34h13m43.540750418s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845319241 (34h13m43.541167994s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845683333 (34h13m43.541532086s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845754208 (34h13m43.541602961s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.846188546 (34h13m43.542037289s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:13.235177054 (34h13m44.931025807s) [upload=30028]: Next returned nil segment Oct 31 21:54:13.235196149 (34h13m44.931044902s) [upload=30028]: Waiting for error group managing segments... Oct 31 21:54:13.731846523 (34h13m45.427695276s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.746649450 (34h13m45.442498203s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.749172355 (34h13m45.445021108s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.824326073 (34h13m45.520174826s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.838257411 (34h13m45.534106154s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.846435784 (34h13m45.542284527s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:14.712603264 (34h13m46.408452007s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:45.241231995 (34h14m16.937080748s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:47.222044337 (34h14m18.91789309s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:57.928194315 (34h14m29.624043068s) [noise=true , address=77.90.31.15:4719 , piece_id=QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:07.960970275 (34h14m39.656819028s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:10.903651570 (34h14m42.599500323s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1YZmFUbHJ3DfhJ1fCqMcy2tdDvmdPogiNAiFwduvcDgsw1QAZQ, address:45.91.116.10:4106): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.903817216 (34h14m42.599665969s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection Oct 31 21:55:10.903935131 (34h14m42.599783884s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection Oct 31 21:55:10.904041113 (34h14m42.599889866s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection Oct 31 21:55:10.904246463 (34h14m42.600095216s) [noise=true , address=77.90.31.14:4674 , piece_id=62IZARJIWZYWAKAVHOP32MZUSTE35VBMA46K472HY44SFP63G5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, address:77.90.31.14:4674): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904369097 (34h14m42.60021785s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection Oct 31 21:55:10.904515886 (34h14m42.600364639s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904544992 (34h14m42.600393735s) [seg_pos={0 0}, upload=30028]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:55:11.152804097 (34h14m42.84865284s) [upload=30028]: Done uploading segments... Oct 31 21:55:11.152854594 (34h14m42.848703337s) [upload=30028]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 8.110621641s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 7.39487706s trace: https://jaeger.dev.storj.io/trace/16457411f40d1a7a FAIL (2025-10-31 22:16:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.394928277s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 14.925389531s trace: https://jaeger.dev.storj.io/trace/4a436e8f0e3652a9 FAIL (2025-10-31 22:16:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== =QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841512922 (34h13m43.537361665s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841695811 (34h13m43.537544554s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842121341 (34h13m43.537970084s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842258863 (34h13m43.538107606s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842653173 (34h13m43.538501917s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843123740 (34h13m43.538972493s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843233278 (34h13m43.539082031s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843685530 (34h13m43.539534283s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844090440 (34h13m43.539939193s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844193336 (34h13m43.540042089s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844586345 (34h13m43.540435098s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844901665 (34h13m43.540750418s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845319241 (34h13m43.541167994s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845683333 (34h13m43.541532086s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845754208 (34h13m43.541602961s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.846188546 (34h13m43.542037289s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:13.235177054 (34h13m44.931025807s) [upload=30028]: Next returned nil segment Oct 31 21:54:13.235196149 (34h13m44.931044902s) [upload=30028]: Waiting for error group managing segments... Oct 31 21:54:13.731846523 (34h13m45.427695276s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.746649450 (34h13m45.442498203s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.749172355 (34h13m45.445021108s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.824326073 (34h13m45.520174826s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.838257411 (34h13m45.534106154s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.846435784 (34h13m45.542284527s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:14.712603264 (34h13m46.408452007s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:45.241231995 (34h14m16.937080748s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:47.222044337 (34h14m18.91789309s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:57.928194315 (34h14m29.624043068s) [noise=true , address=77.90.31.15:4719 , piece_id=QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:07.960970275 (34h14m39.656819028s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:10.903651570 (34h14m42.599500323s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1YZmFUbHJ3DfhJ1fCqMcy2tdDvmdPogiNAiFwduvcDgsw1QAZQ, address:45.91.116.10:4106): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.903817216 (34h14m42.599665969s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection Oct 31 21:55:10.903935131 (34h14m42.599783884s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection Oct 31 21:55:10.904041113 (34h14m42.599889866s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection Oct 31 21:55:10.904246463 (34h14m42.600095216s) [noise=true , address=77.90.31.14:4674 , piece_id=62IZARJIWZYWAKAVHOP32MZUSTE35VBMA46K472HY44SFP63G5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, address:77.90.31.14:4674): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904369097 (34h14m42.60021785s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection Oct 31 21:55:10.904515886 (34h14m42.600364639s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904544992 (34h14m42.600393735s) [seg_pos={0 0}, upload=30028]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:55:11.152804097 (34h14m42.84865284s) [upload=30028]: Done uploading segments... Oct 31 21:55:11.152854594 (34h14m42.848703337s) [upload=30028]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 14.925422114s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................XX..............................] duration: 9.101784823s trace: https://jaeger.dev.storj.io/trace/777326af11170570 FAIL (2025-10-31 22:16:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:54:10.455403616 (34h13m42.151252369s) [upload=30026]: Starting upload Oct 31 21:54:10.455522532 (34h13m42.151371285s) [upload=30026]: Done starting upload Oct 31 21:54:10.455607515 (34h13m42.151456268s) [upload=30026]: Uploading segments... Oct 31 21:54:10.455903067 (34h13m42.15175182s) [upload=30026]: Got next segment. Inline: false Oct 31 21:54:10.455945959 (34h13m42.151794702s) [upload=30026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:54:10.455953403 (34h13m42.151802146s) [upload=30026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:54:10.703207265 (34h13m42.399056029s) [seg_pos={0 0}, upload=30026]: Begin upload segment... Oct 31 21:54:10.703270286 (34h13m42.399119039s) [seg_pos={0 0}, upload=30026]: Done begin upload segment. Oct 31 21:54:10.703295072 (34h13m42.399143825s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703311003 (34h13m42.399159756s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703452613 (34h13m42.399301366s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704249731 (34h13m42.400098484s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704696551 (34h13m42.400545305s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704999748 (34h13m42.400848531s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705356077 (34h13m42.40120483s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705741331 (34h13m42.401590074s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706164356 (34h13m42.402013109s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706471972 (34h13m42.402320715s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706819254 (34h13m42.402667997s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707176183 (34h13m42.403024936s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707611081 (34h13m42.403459834s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708027884 (34h13m42.403876637s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708460038 (34h13m42.404308781s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708841935 (34h13m42.404690688s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:12.455521019 (34h13m44.151369772s) [upload=30026]: Next returned nil segment Oct 31 21:54:12.455539354 (34h13m44.151388107s) [upload=30026]: Waiting for error group managing segments... Oct 31 21:54:12.740273646 (34h13m44.436122389s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742133407 (34h13m44.43798216s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742843700 (34h13m44.438692443s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.743356016 (34h13m44.439204769s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.747176791 (34h13m44.443025535s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.749287941 (34h13m44.445136684s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.751847415 (34h13m44.447696168s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.754964321 (34h13m44.450813084s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.760193528 (34h13m44.456042281s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.763852014 (34h13m44.459700757s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.804034039 (34h13m44.499882792s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:39.766396977 (34h14m11.46224573s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection Oct 31 21:54:39.766509131 (34h14m11.462357874s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:45860->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766523238 (34h14m11.462371981s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 188.166.225.170:49310->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766612227 (34h14m11.46246097s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:52980->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766670488 (34h14m11.462519241s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 188.166.225.170:49096->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766695265 (34h14m11.462544008s) [seg_pos={0 0}, upload=30026]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:54:40.008340748 (34h14m11.704189501s) [upload=30026]: Done uploading segments... Oct 31 21:54:40.008374502 (34h14m11.704223246s) [upload=30026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 9.101830561s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................................X........................X..] duration: 3.765480958s trace: https://jaeger.dev.storj.io/trace/150d6d8439e49137 FAIL (2025-10-31 22:16:26 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.765508119s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.323949963s trace: https://jaeger.dev.storj.io/trace/20f1ef8c79d216e5 FAIL (2025-10-31 22:16:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.323981683s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........................X........................X.XXXXX.] duration: 27.781245663s trace: https://jaeger.dev.storj.io/trace/1efdd793012d80b3 FAIL (2025-10-31 22:16:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.781268606s edge gateway us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.......X.......] duration: 52.185971ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/1ed0d71156ecf368 FAIL (2025-10-31 22:16:10 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:49584->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 52.216658ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................XX.............................] duration: 10.145826241s trace: https://jaeger.dev.storj.io/trace/777326af11170570 FAIL (2025-10-31 22:15:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:54:10.455403616 (34h13m42.151252369s) [upload=30026]: Starting upload Oct 31 21:54:10.455522532 (34h13m42.151371285s) [upload=30026]: Done starting upload Oct 31 21:54:10.455607515 (34h13m42.151456268s) [upload=30026]: Uploading segments... Oct 31 21:54:10.455903067 (34h13m42.15175182s) [upload=30026]: Got next segment. Inline: false Oct 31 21:54:10.455945959 (34h13m42.151794702s) [upload=30026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:54:10.455953403 (34h13m42.151802146s) [upload=30026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:54:10.703207265 (34h13m42.399056029s) [seg_pos={0 0}, upload=30026]: Begin upload segment... Oct 31 21:54:10.703270286 (34h13m42.399119039s) [seg_pos={0 0}, upload=30026]: Done begin upload segment. Oct 31 21:54:10.703295072 (34h13m42.399143825s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703311003 (34h13m42.399159756s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703452613 (34h13m42.399301366s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704249731 (34h13m42.400098484s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704696551 (34h13m42.400545305s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704999748 (34h13m42.400848531s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705356077 (34h13m42.40120483s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705741331 (34h13m42.401590074s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706164356 (34h13m42.402013109s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706471972 (34h13m42.402320715s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706819254 (34h13m42.402667997s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707176183 (34h13m42.403024936s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707611081 (34h13m42.403459834s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708027884 (34h13m42.403876637s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708460038 (34h13m42.404308781s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708841935 (34h13m42.404690688s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:12.455521019 (34h13m44.151369772s) [upload=30026]: Next returned nil segment Oct 31 21:54:12.455539354 (34h13m44.151388107s) [upload=30026]: Waiting for error group managing segments... Oct 31 21:54:12.740273646 (34h13m44.436122389s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742133407 (34h13m44.43798216s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742843700 (34h13m44.438692443s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.743356016 (34h13m44.439204769s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.747176791 (34h13m44.443025535s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.749287941 (34h13m44.445136684s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.751847415 (34h13m44.447696168s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.754964321 (34h13m44.450813084s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.760193528 (34h13m44.456042281s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.763852014 (34h13m44.459700757s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.804034039 (34h13m44.499882792s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:39.766396977 (34h14m11.46224573s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection Oct 31 21:54:39.766509131 (34h14m11.462357874s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:45860->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766523238 (34h14m11.462371981s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 188.166.225.170:49310->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766612227 (34h14m11.46246097s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:52980->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766670488 (34h14m11.462519241s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 188.166.225.170:49096->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766695265 (34h14m11.462544008s) [seg_pos={0 0}, upload=30026]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:54:40.008340748 (34h14m11.704189501s) [upload=30026]: Done uploading segments... Oct 31 21:54:40.008374502 (34h14m11.704223246s) [upload=30026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 10.145856128s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................................X........................X.] duration: 4.200150624s trace: https://jaeger.dev.storj.io/trace/150d6d8439e49137 FAIL (2025-10-31 22:15:45 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.200177316s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [............................X........................X.XXXXX] duration: 30.00142858s trace: https://jaeger.dev.storj.io/trace/1efdd793012d80b3 FAIL (2025-10-31 22:15:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001469478s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.351602352s trace: https://jaeger.dev.storj.io/trace/20f1ef8c79d216e5 FAIL (2025-10-31 22:15:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.351634782s edge gateway us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [........X......] duration: 45.099389ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/1ed0d71156ecf368 FAIL (2025-10-31 22:15:40 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:49584->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 45.126571ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 7.793051582s trace: https://jaeger.dev.storj.io/trace/16457411f40d1a7a FAIL (2025-10-31 22:15:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.793079514s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................XX............................] duration: 21.786880245s trace: https://jaeger.dev.storj.io/trace/777326af11170570 FAIL (2025-10-31 22:15:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:54:10.455403616 (34h13m42.151252369s) [upload=30026]: Starting upload Oct 31 21:54:10.455522532 (34h13m42.151371285s) [upload=30026]: Done starting upload Oct 31 21:54:10.455607515 (34h13m42.151456268s) [upload=30026]: Uploading segments... Oct 31 21:54:10.455903067 (34h13m42.15175182s) [upload=30026]: Got next segment. Inline: false Oct 31 21:54:10.455945959 (34h13m42.151794702s) [upload=30026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:54:10.455953403 (34h13m42.151802146s) [upload=30026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:54:10.703207265 (34h13m42.399056029s) [seg_pos={0 0}, upload=30026]: Begin upload segment... Oct 31 21:54:10.703270286 (34h13m42.399119039s) [seg_pos={0 0}, upload=30026]: Done begin upload segment. Oct 31 21:54:10.703295072 (34h13m42.399143825s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703311003 (34h13m42.399159756s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703452613 (34h13m42.399301366s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704249731 (34h13m42.400098484s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704696551 (34h13m42.400545305s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704999748 (34h13m42.400848531s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705356077 (34h13m42.40120483s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705741331 (34h13m42.401590074s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706164356 (34h13m42.402013109s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706471972 (34h13m42.402320715s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706819254 (34h13m42.402667997s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707176183 (34h13m42.403024936s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707611081 (34h13m42.403459834s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708027884 (34h13m42.403876637s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708460038 (34h13m42.404308781s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708841935 (34h13m42.404690688s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:12.455521019 (34h13m44.151369772s) [upload=30026]: Next returned nil segment Oct 31 21:54:12.455539354 (34h13m44.151388107s) [upload=30026]: Waiting for error group managing segments... Oct 31 21:54:12.740273646 (34h13m44.436122389s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742133407 (34h13m44.43798216s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742843700 (34h13m44.438692443s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.743356016 (34h13m44.439204769s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.747176791 (34h13m44.443025535s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.749287941 (34h13m44.445136684s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.751847415 (34h13m44.447696168s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.754964321 (34h13m44.450813084s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.760193528 (34h13m44.456042281s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.763852014 (34h13m44.459700757s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.804034039 (34h13m44.499882792s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:39.766396977 (34h14m11.46224573s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection Oct 31 21:54:39.766509131 (34h14m11.462357874s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:45860->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766523238 (34h14m11.462371981s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 188.166.225.170:49310->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766612227 (34h14m11.46246097s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:52980->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766670488 (34h14m11.462519241s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 188.166.225.170:49096->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766695265 (34h14m11.462544008s) [seg_pos={0 0}, upload=30026]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:54:40.008340748 (34h14m11.704189501s) [upload=30026]: Done uploading segments... Oct 31 21:54:40.008374502 (34h14m11.704223246s) [upload=30026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 21.786902167s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 8.965115662s trace: https://jaeger.dev.storj.io/trace/4a436e8f0e3652a9 FAIL (2025-10-31 22:15:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== =QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841512922 (34h13m43.537361665s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841695811 (34h13m43.537544554s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842121341 (34h13m43.537970084s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842258863 (34h13m43.538107606s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842653173 (34h13m43.538501917s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843123740 (34h13m43.538972493s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843233278 (34h13m43.539082031s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843685530 (34h13m43.539534283s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844090440 (34h13m43.539939193s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844193336 (34h13m43.540042089s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844586345 (34h13m43.540435098s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844901665 (34h13m43.540750418s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845319241 (34h13m43.541167994s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845683333 (34h13m43.541532086s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845754208 (34h13m43.541602961s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.846188546 (34h13m43.542037289s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:13.235177054 (34h13m44.931025807s) [upload=30028]: Next returned nil segment Oct 31 21:54:13.235196149 (34h13m44.931044902s) [upload=30028]: Waiting for error group managing segments... Oct 31 21:54:13.731846523 (34h13m45.427695276s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.746649450 (34h13m45.442498203s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.749172355 (34h13m45.445021108s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.824326073 (34h13m45.520174826s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.838257411 (34h13m45.534106154s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.846435784 (34h13m45.542284527s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:14.712603264 (34h13m46.408452007s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:45.241231995 (34h14m16.937080748s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:47.222044337 (34h14m18.91789309s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:57.928194315 (34h14m29.624043068s) [noise=true , address=77.90.31.15:4719 , piece_id=QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:07.960970275 (34h14m39.656819028s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:10.903651570 (34h14m42.599500323s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1YZmFUbHJ3DfhJ1fCqMcy2tdDvmdPogiNAiFwduvcDgsw1QAZQ, address:45.91.116.10:4106): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.903817216 (34h14m42.599665969s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection Oct 31 21:55:10.903935131 (34h14m42.599783884s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection Oct 31 21:55:10.904041113 (34h14m42.599889866s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection Oct 31 21:55:10.904246463 (34h14m42.600095216s) [noise=true , address=77.90.31.14:4674 , piece_id=62IZARJIWZYWAKAVHOP32MZUSTE35VBMA46K472HY44SFP63G5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, address:77.90.31.14:4674): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904369097 (34h14m42.60021785s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection Oct 31 21:55:10.904515886 (34h14m42.600364639s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904544992 (34h14m42.600393735s) [seg_pos={0 0}, upload=30028]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:55:11.152804097 (34h14m42.84865284s) [upload=30028]: Done uploading segments... Oct 31 21:55:11.152854594 (34h14m42.848703337s) [upload=30028]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 8.96513997s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.37456232s trace: https://jaeger.dev.storj.io/trace/20f1ef8c79d216e5 FAIL (2025-10-31 22:15:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.374594211s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............................X........................X.XXXX] duration: 30.001666706s trace: https://jaeger.dev.storj.io/trace/434ebe0d12e8b203 FAIL (2025-10-31 22:15:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001692224s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................................X........................X] duration: 5.00202488s trace: https://jaeger.dev.storj.io/trace/150d6d8439e49137 FAIL (2025-10-31 22:15:05 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002049035s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.........X.....] duration: 53.958795ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/1ed0d71156ecf368 FAIL (2025-10-31 22:14:57 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:49584->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 53.988131ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 13.951684013s trace: https://jaeger.dev.storj.io/trace/4a436e8f0e3652a9 FAIL (2025-10-31 22:14:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== =QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841512922 (34h13m43.537361665s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841695811 (34h13m43.537544554s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842121341 (34h13m43.537970084s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842258863 (34h13m43.538107606s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842653173 (34h13m43.538501917s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843123740 (34h13m43.538972493s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843233278 (34h13m43.539082031s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843685530 (34h13m43.539534283s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844090440 (34h13m43.539939193s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844193336 (34h13m43.540042089s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844586345 (34h13m43.540435098s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844901665 (34h13m43.540750418s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845319241 (34h13m43.541167994s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845683333 (34h13m43.541532086s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845754208 (34h13m43.541602961s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.846188546 (34h13m43.542037289s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:13.235177054 (34h13m44.931025807s) [upload=30028]: Next returned nil segment Oct 31 21:54:13.235196149 (34h13m44.931044902s) [upload=30028]: Waiting for error group managing segments... Oct 31 21:54:13.731846523 (34h13m45.427695276s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.746649450 (34h13m45.442498203s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.749172355 (34h13m45.445021108s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.824326073 (34h13m45.520174826s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.838257411 (34h13m45.534106154s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.846435784 (34h13m45.542284527s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:14.712603264 (34h13m46.408452007s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:45.241231995 (34h14m16.937080748s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:47.222044337 (34h14m18.91789309s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:57.928194315 (34h14m29.624043068s) [noise=true , address=77.90.31.15:4719 , piece_id=QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:07.960970275 (34h14m39.656819028s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:10.903651570 (34h14m42.599500323s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1YZmFUbHJ3DfhJ1fCqMcy2tdDvmdPogiNAiFwduvcDgsw1QAZQ, address:45.91.116.10:4106): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.903817216 (34h14m42.599665969s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection Oct 31 21:55:10.903935131 (34h14m42.599783884s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection Oct 31 21:55:10.904041113 (34h14m42.599889866s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection Oct 31 21:55:10.904246463 (34h14m42.600095216s) [noise=true , address=77.90.31.14:4674 , piece_id=62IZARJIWZYWAKAVHOP32MZUSTE35VBMA46K472HY44SFP63G5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, address:77.90.31.14:4674): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904369097 (34h14m42.60021785s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection Oct 31 21:55:10.904515886 (34h14m42.600364639s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904544992 (34h14m42.600393735s) [seg_pos={0 0}, upload=30028]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:55:11.152804097 (34h14m42.84865284s) [upload=30028]: Done uploading segments... Oct 31 21:55:11.152854594 (34h14m42.848703337s) [upload=30028]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 13.951704662s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.325295557s trace: https://jaeger.dev.storj.io/trace/20f1ef8c79d216e5 FAIL (2025-10-31 22:14:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.325340955s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............................X........................X.XXX] duration: 30.001374901s trace: https://jaeger.dev.storj.io/trace/1e1f88af716fc9a0 FAIL (2025-10-31 22:14:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001404817s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 7.387932012s trace: https://jaeger.dev.storj.io/trace/16457411f40d1a7a FAIL (2025-10-31 22:14:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.38795628s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 16.699659883s trace: https://jaeger.dev.storj.io/trace/4a436e8f0e3652a9 FAIL (2025-10-31 22:14:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== =QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841512922 (34h13m43.537361665s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841695811 (34h13m43.537544554s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842121341 (34h13m43.537970084s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842258863 (34h13m43.538107606s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842653173 (34h13m43.538501917s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843123740 (34h13m43.538972493s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843233278 (34h13m43.539082031s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843685530 (34h13m43.539534283s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844090440 (34h13m43.539939193s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844193336 (34h13m43.540042089s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844586345 (34h13m43.540435098s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844901665 (34h13m43.540750418s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845319241 (34h13m43.541167994s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845683333 (34h13m43.541532086s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845754208 (34h13m43.541602961s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.846188546 (34h13m43.542037289s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:13.235177054 (34h13m44.931025807s) [upload=30028]: Next returned nil segment Oct 31 21:54:13.235196149 (34h13m44.931044902s) [upload=30028]: Waiting for error group managing segments... Oct 31 21:54:13.731846523 (34h13m45.427695276s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.746649450 (34h13m45.442498203s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.749172355 (34h13m45.445021108s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.824326073 (34h13m45.520174826s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.838257411 (34h13m45.534106154s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.846435784 (34h13m45.542284527s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:14.712603264 (34h13m46.408452007s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:45.241231995 (34h14m16.937080748s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:47.222044337 (34h14m18.91789309s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:57.928194315 (34h14m29.624043068s) [noise=true , address=77.90.31.15:4719 , piece_id=QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:07.960970275 (34h14m39.656819028s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:10.903651570 (34h14m42.599500323s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1YZmFUbHJ3DfhJ1fCqMcy2tdDvmdPogiNAiFwduvcDgsw1QAZQ, address:45.91.116.10:4106): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.903817216 (34h14m42.599665969s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection Oct 31 21:55:10.903935131 (34h14m42.599783884s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection Oct 31 21:55:10.904041113 (34h14m42.599889866s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection Oct 31 21:55:10.904246463 (34h14m42.600095216s) [noise=true , address=77.90.31.14:4674 , piece_id=62IZARJIWZYWAKAVHOP32MZUSTE35VBMA46K472HY44SFP63G5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, address:77.90.31.14:4674): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904369097 (34h14m42.60021785s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection Oct 31 21:55:10.904515886 (34h14m42.600364639s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904544992 (34h14m42.600393735s) [seg_pos={0 0}, upload=30028]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:55:11.152804097 (34h14m42.84865284s) [upload=30028]: Done uploading segments... Oct 31 21:55:11.152854594 (34h14m42.848703337s) [upload=30028]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 16.699694591s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..........X....] duration: 66.967966ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/1ed0d71156ecf368 FAIL (2025-10-31 22:14:18 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:49584->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 66.999877ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................X........................] duration: 5.00051434s trace: https://jaeger.dev.storj.io/trace/74d5301c9b50695d FAIL (2025-10-31 22:14:10 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000544499s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.331040006s trace: https://jaeger.dev.storj.io/trace/20f1ef8c79d216e5 FAIL (2025-10-31 22:14:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.331065855s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................XX...........................] duration: 10.021066565s trace: https://jaeger.dev.storj.io/trace/777326af11170570 FAIL (2025-10-31 22:14:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:54:10.455403616 (34h13m42.151252369s) [upload=30026]: Starting upload Oct 31 21:54:10.455522532 (34h13m42.151371285s) [upload=30026]: Done starting upload Oct 31 21:54:10.455607515 (34h13m42.151456268s) [upload=30026]: Uploading segments... Oct 31 21:54:10.455903067 (34h13m42.15175182s) [upload=30026]: Got next segment. Inline: false Oct 31 21:54:10.455945959 (34h13m42.151794702s) [upload=30026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:54:10.455953403 (34h13m42.151802146s) [upload=30026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:54:10.703207265 (34h13m42.399056029s) [seg_pos={0 0}, upload=30026]: Begin upload segment... Oct 31 21:54:10.703270286 (34h13m42.399119039s) [seg_pos={0 0}, upload=30026]: Done begin upload segment. Oct 31 21:54:10.703295072 (34h13m42.399143825s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703311003 (34h13m42.399159756s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703452613 (34h13m42.399301366s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704249731 (34h13m42.400098484s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704696551 (34h13m42.400545305s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704999748 (34h13m42.400848531s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705356077 (34h13m42.40120483s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705741331 (34h13m42.401590074s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706164356 (34h13m42.402013109s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706471972 (34h13m42.402320715s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706819254 (34h13m42.402667997s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707176183 (34h13m42.403024936s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707611081 (34h13m42.403459834s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708027884 (34h13m42.403876637s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708460038 (34h13m42.404308781s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708841935 (34h13m42.404690688s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:12.455521019 (34h13m44.151369772s) [upload=30026]: Next returned nil segment Oct 31 21:54:12.455539354 (34h13m44.151388107s) [upload=30026]: Waiting for error group managing segments... Oct 31 21:54:12.740273646 (34h13m44.436122389s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742133407 (34h13m44.43798216s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742843700 (34h13m44.438692443s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.743356016 (34h13m44.439204769s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.747176791 (34h13m44.443025535s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.749287941 (34h13m44.445136684s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.751847415 (34h13m44.447696168s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.754964321 (34h13m44.450813084s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.760193528 (34h13m44.456042281s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.763852014 (34h13m44.459700757s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.804034039 (34h13m44.499882792s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:39.766396977 (34h14m11.46224573s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection Oct 31 21:54:39.766509131 (34h14m11.462357874s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:45860->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766523238 (34h14m11.462371981s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 188.166.225.170:49310->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766612227 (34h14m11.46246097s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:52980->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766670488 (34h14m11.462519241s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 188.166.225.170:49096->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766695265 (34h14m11.462544008s) [seg_pos={0 0}, upload=30026]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:54:40.008340748 (34h14m11.704189501s) [upload=30026]: Done uploading segments... Oct 31 21:54:40.008374502 (34h14m11.704223246s) [upload=30026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 10.021089328s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 9.280158632s trace: https://jaeger.dev.storj.io/trace/4a436e8f0e3652a9 FAIL (2025-10-31 22:13:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== =QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841512922 (34h13m43.537361665s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841695811 (34h13m43.537544554s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842121341 (34h13m43.537970084s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842258863 (34h13m43.538107606s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842653173 (34h13m43.538501917s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843123740 (34h13m43.538972493s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843233278 (34h13m43.539082031s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843685530 (34h13m43.539534283s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844090440 (34h13m43.539939193s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844193336 (34h13m43.540042089s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844586345 (34h13m43.540435098s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844901665 (34h13m43.540750418s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845319241 (34h13m43.541167994s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845683333 (34h13m43.541532086s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845754208 (34h13m43.541602961s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.846188546 (34h13m43.542037289s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:13.235177054 (34h13m44.931025807s) [upload=30028]: Next returned nil segment Oct 31 21:54:13.235196149 (34h13m44.931044902s) [upload=30028]: Waiting for error group managing segments... Oct 31 21:54:13.731846523 (34h13m45.427695276s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.746649450 (34h13m45.442498203s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.749172355 (34h13m45.445021108s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.824326073 (34h13m45.520174826s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.838257411 (34h13m45.534106154s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.846435784 (34h13m45.542284527s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:14.712603264 (34h13m46.408452007s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:45.241231995 (34h14m16.937080748s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:47.222044337 (34h14m18.91789309s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:57.928194315 (34h14m29.624043068s) [noise=true , address=77.90.31.15:4719 , piece_id=QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:07.960970275 (34h14m39.656819028s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:10.903651570 (34h14m42.599500323s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1YZmFUbHJ3DfhJ1fCqMcy2tdDvmdPogiNAiFwduvcDgsw1QAZQ, address:45.91.116.10:4106): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.903817216 (34h14m42.599665969s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection Oct 31 21:55:10.903935131 (34h14m42.599783884s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection Oct 31 21:55:10.904041113 (34h14m42.599889866s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection Oct 31 21:55:10.904246463 (34h14m42.600095216s) [noise=true , address=77.90.31.14:4674 , piece_id=62IZARJIWZYWAKAVHOP32MZUSTE35VBMA46K472HY44SFP63G5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, address:77.90.31.14:4674): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904369097 (34h14m42.60021785s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection Oct 31 21:55:10.904515886 (34h14m42.600364639s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904544992 (34h14m42.600393735s) [seg_pos={0 0}, upload=30028]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:55:11.152804097 (34h14m42.84865284s) [upload=30028]: Done uploading segments... Oct 31 21:55:11.152854594 (34h14m42.848703337s) [upload=30028]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 9.280191284s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............................X........................X.XX] duration: 30.001794857s trace: https://jaeger.dev.storj.io/trace/71eb1deb7e9c7a2d FAIL (2025-10-31 22:13:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00195967s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 8.279956141s trace: https://jaeger.dev.storj.io/trace/16457411f40d1a7a FAIL (2025-10-31 22:13:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.280006167s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [X...................................X.......................] duration: 4.182386267s trace: https://jaeger.dev.storj.io/trace/74d5301c9b50695d FAIL (2025-10-31 22:13:45 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.18241438s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................XX..........................] duration: 16.766775557s trace: https://jaeger.dev.storj.io/trace/777326af11170570 FAIL (2025-10-31 22:13:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:54:10.455403616 (34h13m42.151252369s) [upload=30026]: Starting upload Oct 31 21:54:10.455522532 (34h13m42.151371285s) [upload=30026]: Done starting upload Oct 31 21:54:10.455607515 (34h13m42.151456268s) [upload=30026]: Uploading segments... Oct 31 21:54:10.455903067 (34h13m42.15175182s) [upload=30026]: Got next segment. Inline: false Oct 31 21:54:10.455945959 (34h13m42.151794702s) [upload=30026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:54:10.455953403 (34h13m42.151802146s) [upload=30026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:54:10.703207265 (34h13m42.399056029s) [seg_pos={0 0}, upload=30026]: Begin upload segment... Oct 31 21:54:10.703270286 (34h13m42.399119039s) [seg_pos={0 0}, upload=30026]: Done begin upload segment. Oct 31 21:54:10.703295072 (34h13m42.399143825s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703311003 (34h13m42.399159756s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703452613 (34h13m42.399301366s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704249731 (34h13m42.400098484s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704696551 (34h13m42.400545305s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704999748 (34h13m42.400848531s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705356077 (34h13m42.40120483s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705741331 (34h13m42.401590074s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706164356 (34h13m42.402013109s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706471972 (34h13m42.402320715s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706819254 (34h13m42.402667997s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707176183 (34h13m42.403024936s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707611081 (34h13m42.403459834s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708027884 (34h13m42.403876637s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708460038 (34h13m42.404308781s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708841935 (34h13m42.404690688s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:12.455521019 (34h13m44.151369772s) [upload=30026]: Next returned nil segment Oct 31 21:54:12.455539354 (34h13m44.151388107s) [upload=30026]: Waiting for error group managing segments... Oct 31 21:54:12.740273646 (34h13m44.436122389s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742133407 (34h13m44.43798216s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742843700 (34h13m44.438692443s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.743356016 (34h13m44.439204769s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.747176791 (34h13m44.443025535s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.749287941 (34h13m44.445136684s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.751847415 (34h13m44.447696168s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.754964321 (34h13m44.450813084s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.760193528 (34h13m44.456042281s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.763852014 (34h13m44.459700757s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.804034039 (34h13m44.499882792s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:39.766396977 (34h14m11.46224573s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection Oct 31 21:54:39.766509131 (34h14m11.462357874s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:45860->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766523238 (34h14m11.462371981s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 188.166.225.170:49310->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766612227 (34h14m11.46246097s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:52980->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766670488 (34h14m11.462519241s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 188.166.225.170:49096->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766695265 (34h14m11.462544008s) [seg_pos={0 0}, upload=30026]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:54:40.008340748 (34h14m11.704189501s) [upload=30026]: Done uploading segments... Oct 31 21:54:40.008374502 (34h14m11.704223246s) [upload=30026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 16.766805655s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.307397249s trace: https://jaeger.dev.storj.io/trace/20f1ef8c79d216e5 FAIL (2025-10-31 22:13:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.307439008s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 8.289403916s trace: https://jaeger.dev.storj.io/trace/16457411f40d1a7a FAIL (2025-10-31 22:13:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.289434954s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [...........X...] duration: 46.213157ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/1ed0d71156ecf368 FAIL (2025-10-31 22:13:14 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:49584->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 46.276888ms core website
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................X........................X.X] duration: 30.001822044s trace: https://jaeger.dev.storj.io/trace/503f16a3abebc273 FAIL (2025-10-31 22:13:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001873322s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.041354596s trace: https://jaeger.dev.storj.io/trace/16457411f40d1a7a FAIL (2025-10-31 22:12:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.041387218s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.................................X.........................] duration: 8.690785278s trace: https://jaeger.dev.storj.io/trace/4a436e8f0e3652a9 FAIL (2025-10-31 22:12:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== =QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841512922 (34h13m43.537361665s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841695811 (34h13m43.537544554s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842121341 (34h13m43.537970084s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842258863 (34h13m43.538107606s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842653173 (34h13m43.538501917s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843123740 (34h13m43.538972493s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843233278 (34h13m43.539082031s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843685530 (34h13m43.539534283s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844090440 (34h13m43.539939193s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844193336 (34h13m43.540042089s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844586345 (34h13m43.540435098s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844901665 (34h13m43.540750418s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845319241 (34h13m43.541167994s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845683333 (34h13m43.541532086s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845754208 (34h13m43.541602961s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.846188546 (34h13m43.542037289s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:13.235177054 (34h13m44.931025807s) [upload=30028]: Next returned nil segment Oct 31 21:54:13.235196149 (34h13m44.931044902s) [upload=30028]: Waiting for error group managing segments... Oct 31 21:54:13.731846523 (34h13m45.427695276s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.746649450 (34h13m45.442498203s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.749172355 (34h13m45.445021108s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.824326073 (34h13m45.520174826s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.838257411 (34h13m45.534106154s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.846435784 (34h13m45.542284527s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:14.712603264 (34h13m46.408452007s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:45.241231995 (34h14m16.937080748s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:47.222044337 (34h14m18.91789309s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:57.928194315 (34h14m29.624043068s) [noise=true , address=77.90.31.15:4719 , piece_id=QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:07.960970275 (34h14m39.656819028s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:10.903651570 (34h14m42.599500323s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1YZmFUbHJ3DfhJ1fCqMcy2tdDvmdPogiNAiFwduvcDgsw1QAZQ, address:45.91.116.10:4106): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.903817216 (34h14m42.599665969s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection Oct 31 21:55:10.903935131 (34h14m42.599783884s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection Oct 31 21:55:10.904041113 (34h14m42.599889866s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection Oct 31 21:55:10.904246463 (34h14m42.600095216s) [noise=true , address=77.90.31.14:4674 , piece_id=62IZARJIWZYWAKAVHOP32MZUSTE35VBMA46K472HY44SFP63G5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, address:77.90.31.14:4674): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904369097 (34h14m42.60021785s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection Oct 31 21:55:10.904515886 (34h14m42.600364639s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904544992 (34h14m42.600393735s) [seg_pos={0 0}, upload=30028]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:55:11.152804097 (34h14m42.84865284s) [upload=30028]: Done uploading segments... Oct 31 21:55:11.152854594 (34h14m42.848703337s) [upload=30028]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 8.690815856s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001486439s trace: https://jaeger.dev.storj.io/trace/20f1ef8c79d216e5 FAIL (2025-10-31 22:12:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00151834s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.X...................................X......................] duration: 3.587102768s trace: https://jaeger.dev.storj.io/trace/74d5301c9b50695d FAIL (2025-10-31 22:12:32 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.58712986s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [............X..] duration: 59.869039ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/1ed0d71156ecf368 FAIL (2025-10-31 22:12:23 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:49584->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 59.886702ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X................................XX.........................] duration: 6.180098473s trace: https://jaeger.dev.storj.io/trace/777326af11170570 FAIL (2025-10-31 22:12:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:54:10.455403616 (34h13m42.151252369s) [upload=30026]: Starting upload Oct 31 21:54:10.455522532 (34h13m42.151371285s) [upload=30026]: Done starting upload Oct 31 21:54:10.455607515 (34h13m42.151456268s) [upload=30026]: Uploading segments... Oct 31 21:54:10.455903067 (34h13m42.15175182s) [upload=30026]: Got next segment. Inline: false Oct 31 21:54:10.455945959 (34h13m42.151794702s) [upload=30026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:54:10.455953403 (34h13m42.151802146s) [upload=30026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:54:10.703207265 (34h13m42.399056029s) [seg_pos={0 0}, upload=30026]: Begin upload segment... Oct 31 21:54:10.703270286 (34h13m42.399119039s) [seg_pos={0 0}, upload=30026]: Done begin upload segment. Oct 31 21:54:10.703295072 (34h13m42.399143825s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703311003 (34h13m42.399159756s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703452613 (34h13m42.399301366s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704249731 (34h13m42.400098484s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704696551 (34h13m42.400545305s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704999748 (34h13m42.400848531s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705356077 (34h13m42.40120483s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705741331 (34h13m42.401590074s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706164356 (34h13m42.402013109s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706471972 (34h13m42.402320715s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706819254 (34h13m42.402667997s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707176183 (34h13m42.403024936s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707611081 (34h13m42.403459834s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708027884 (34h13m42.403876637s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708460038 (34h13m42.404308781s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708841935 (34h13m42.404690688s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:12.455521019 (34h13m44.151369772s) [upload=30026]: Next returned nil segment Oct 31 21:54:12.455539354 (34h13m44.151388107s) [upload=30026]: Waiting for error group managing segments... Oct 31 21:54:12.740273646 (34h13m44.436122389s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742133407 (34h13m44.43798216s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742843700 (34h13m44.438692443s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.743356016 (34h13m44.439204769s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.747176791 (34h13m44.443025535s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.749287941 (34h13m44.445136684s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.751847415 (34h13m44.447696168s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.754964321 (34h13m44.450813084s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.760193528 (34h13m44.456042281s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.763852014 (34h13m44.459700757s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.804034039 (34h13m44.499882792s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:39.766396977 (34h14m11.46224573s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection Oct 31 21:54:39.766509131 (34h14m11.462357874s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:45860->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766523238 (34h14m11.462371981s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 188.166.225.170:49310->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766612227 (34h14m11.46246097s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:52980->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766670488 (34h14m11.462519241s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 188.166.225.170:49096->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766695265 (34h14m11.462544008s) [seg_pos={0 0}, upload=30026]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:54:40.008340748 (34h14m11.704189501s) [upload=30026]: Done uploading segments... Oct 31 21:54:40.008374502 (34h14m11.704223246s) [upload=30026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.18011779s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X........................X.] duration: 6.906376049s trace: https://jaeger.dev.storj.io/trace/88a5b63e7e6903a FAIL (2025-10-31 22:12:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.906407268s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.................................X........................X] duration: 30.000700045s trace: https://jaeger.dev.storj.io/trace/88a5b63e7e6903a FAIL (2025-10-31 22:12:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000731514s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [XX.................................X........................] duration: 9.981660977s trace: https://jaeger.dev.storj.io/trace/4a436e8f0e3652a9 FAIL (2025-10-31 22:11:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== =QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841512922 (34h13m43.537361665s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841695811 (34h13m43.537544554s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842121341 (34h13m43.537970084s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842258863 (34h13m43.538107606s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842653173 (34h13m43.538501917s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843123740 (34h13m43.538972493s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843233278 (34h13m43.539082031s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843685530 (34h13m43.539534283s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844090440 (34h13m43.539939193s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844193336 (34h13m43.540042089s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844586345 (34h13m43.540435098s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844901665 (34h13m43.540750418s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845319241 (34h13m43.541167994s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845683333 (34h13m43.541532086s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845754208 (34h13m43.541602961s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.846188546 (34h13m43.542037289s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:13.235177054 (34h13m44.931025807s) [upload=30028]: Next returned nil segment Oct 31 21:54:13.235196149 (34h13m44.931044902s) [upload=30028]: Waiting for error group managing segments... Oct 31 21:54:13.731846523 (34h13m45.427695276s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.746649450 (34h13m45.442498203s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.749172355 (34h13m45.445021108s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.824326073 (34h13m45.520174826s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.838257411 (34h13m45.534106154s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.846435784 (34h13m45.542284527s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:14.712603264 (34h13m46.408452007s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:45.241231995 (34h14m16.937080748s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:47.222044337 (34h14m18.91789309s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:57.928194315 (34h14m29.624043068s) [noise=true , address=77.90.31.15:4719 , piece_id=QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:07.960970275 (34h14m39.656819028s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:10.903651570 (34h14m42.599500323s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1YZmFUbHJ3DfhJ1fCqMcy2tdDvmdPogiNAiFwduvcDgsw1QAZQ, address:45.91.116.10:4106): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.903817216 (34h14m42.599665969s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection Oct 31 21:55:10.903935131 (34h14m42.599783884s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection Oct 31 21:55:10.904041113 (34h14m42.599889866s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection Oct 31 21:55:10.904246463 (34h14m42.600095216s) [noise=true , address=77.90.31.14:4674 , piece_id=62IZARJIWZYWAKAVHOP32MZUSTE35VBMA46K472HY44SFP63G5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, address:77.90.31.14:4674): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904369097 (34h14m42.60021785s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection Oct 31 21:55:10.904515886 (34h14m42.600364639s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904544992 (34h14m42.600393735s) [seg_pos={0 0}, upload=30028]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:55:11.152804097 (34h14m42.84865284s) [upload=30028]: Done uploading segments... Oct 31 21:55:11.152854594 (34h14m42.848703337s) [upload=30028]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 9.98168859s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.692300386s trace: https://jaeger.dev.storj.io/trace/16457411f40d1a7a FAIL (2025-10-31 22:11:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.6924546s core uplink us-select-1 us1-placement-20

1
...
5553
5554
5555
5556
5557
...
10098