Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [X.................................X........................X] duration:
30.000700045s trace:
https://jaeger.dev.storj.io/trace/88a5b63e7e6903a
FAIL (2025-10-31 22:12:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000731514s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [XX.................................X........................] duration:
9.981660977s trace:
https://jaeger.dev.storj.io/trace/4a436e8f0e3652a9
FAIL (2025-10-31 22:11:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
=QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.841512922 (34h13m43.537361665s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.841695811 (34h13m43.537544554s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.842121341 (34h13m43.537970084s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.842258863 (34h13m43.538107606s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.842653173 (34h13m43.538501917s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.843123740 (34h13m43.538972493s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.843233278 (34h13m43.539082031s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.843685530 (34h13m43.539534283s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.844090440 (34h13m43.539939193s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.844193336 (34h13m43.540042089s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.844586345 (34h13m43.540435098s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.844901665 (34h13m43.540750418s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.845319241 (34h13m43.541167994s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.845683333 (34h13m43.541532086s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.845754208 (34h13m43.541602961s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.846188546 (34h13m43.542037289s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:13.235177054 (34h13m44.931025807s) [upload=30028]: Next returned nil segment
Oct 31 21:54:13.235196149 (34h13m44.931044902s) [upload=30028]: Waiting for error group managing segments...
Oct 31 21:54:13.731846523 (34h13m45.427695276s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.746649450 (34h13m45.442498203s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.749172355 (34h13m45.445021108s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.824326073 (34h13m45.520174826s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.838257411 (34h13m45.534106154s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.846435784 (34h13m45.542284527s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:14.712603264 (34h13m46.408452007s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:45.241231995 (34h14m16.937080748s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:47.222044337 (34h14m18.91789309s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:57.928194315 (34h14m29.624043068s) [noise=true , address=77.90.31.15:4719 , piece_id=QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:55:07.960970275 (34h14m39.656819028s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:55:10.903651570 (34h14m42.599500323s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1YZmFUbHJ3DfhJ1fCqMcy2tdDvmdPogiNAiFwduvcDgsw1QAZQ, address:45.91.116.10:4106): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:55:10.903817216 (34h14m42.599665969s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection
Oct 31 21:55:10.903935131 (34h14m42.599783884s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection
Oct 31 21:55:10.904041113 (34h14m42.599889866s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection
Oct 31 21:55:10.904246463 (34h14m42.600095216s) [noise=true , address=77.90.31.14:4674 , piece_id=62IZARJIWZYWAKAVHOP32MZUSTE35VBMA46K472HY44SFP63G5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, address:77.90.31.14:4674): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:55:10.904369097 (34h14m42.60021785s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection
Oct 31 21:55:10.904515886 (34h14m42.600364639s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:55:10.904544992 (34h14m42.600393735s) [seg_pos={0 0}, upload=30028]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Oct 31 21:55:11.152804097 (34h14m42.84865284s) [upload=30028]: Done uploading segments...
Oct 31 21:55:11.152854594 (34h14m42.848703337s) [upload=30028]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 9.98168859s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
6.692300386s trace:
https://jaeger.dev.storj.io/trace/16457411f40d1a7a
FAIL (2025-10-31 22:11:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.6924546s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)antiblip:
1 out of 15 is failed (threshold=6) [.............X.] duration:
46.693441ms remote:
"tcp:[2620:cb:2000::1]:443" target:
https://www.storj.io/ trace:
https://jaeger.dev.storj.io/trace/1ed0d71156ecf368
FAIL (2025-10-31 22:11:48 UTC)
orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:49584->[2620:cb:2000::1]:443: read: connection reset by peer
Duration: 46.713589ms
core
website
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.X................................XX........................] duration:
6.307770537s trace:
https://jaeger.dev.storj.io/trace/777326af11170570
FAIL (2025-10-31 22:11:44 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 31 21:54:10.455403616 (34h13m42.151252369s) [upload=30026]: Starting upload
Oct 31 21:54:10.455522532 (34h13m42.151371285s) [upload=30026]: Done starting upload
Oct 31 21:54:10.455607515 (34h13m42.151456268s) [upload=30026]: Uploading segments...
Oct 31 21:54:10.455903067 (34h13m42.15175182s) [upload=30026]: Got next segment. Inline: false
Oct 31 21:54:10.455945959 (34h13m42.151794702s) [upload=30026]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:10.455953403 (34h13m42.151802146s) [upload=30026]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:10.703207265 (34h13m42.399056029s) [seg_pos={0 0}, upload=30026]: Begin upload segment...
Oct 31 21:54:10.703270286 (34h13m42.399119039s) [seg_pos={0 0}, upload=30026]: Done begin upload segment.
Oct 31 21:54:10.703295072 (34h13m42.399143825s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.703311003 (34h13m42.399159756s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.703452613 (34h13m42.399301366s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.704249731 (34h13m42.400098484s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.704696551 (34h13m42.400545305s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.704999748 (34h13m42.400848531s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.705356077 (34h13m42.40120483s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.705741331 (34h13m42.401590074s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.706164356 (34h13m42.402013109s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.706471972 (34h13m42.402320715s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.706819254 (34h13m42.402667997s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.707176183 (34h13m42.403024936s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.707611081 (34h13m42.403459834s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.708027884 (34h13m42.403876637s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.708460038 (34h13m42.404308781s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.708841935 (34h13m42.404690688s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:12.455521019 (34h13m44.151369772s) [upload=30026]: Next returned nil segment
Oct 31 21:54:12.455539354 (34h13m44.151388107s) [upload=30026]: Waiting for error group managing segments...
Oct 31 21:54:12.740273646 (34h13m44.436122389s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.742133407 (34h13m44.43798216s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.742843700 (34h13m44.438692443s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.743356016 (34h13m44.439204769s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.747176791 (34h13m44.443025535s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.749287941 (34h13m44.445136684s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.751847415 (34h13m44.447696168s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.754964321 (34h13m44.450813084s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.760193528 (34h13m44.456042281s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.763852014 (34h13m44.459700757s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.804034039 (34h13m44.499882792s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:39.766396977 (34h14m11.46224573s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection
Oct 31 21:54:39.766509131 (34h14m11.462357874s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:45860->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:39.766523238 (34h14m11.462371981s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 188.166.225.170:49310->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:39.766612227 (34h14m11.46246097s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:52980->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:39.766670488 (34h14m11.462519241s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 188.166.225.170:49096->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:39.766695265 (34h14m11.462544008s) [seg_pos={0 0}, upload=30026]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 31 21:54:40.008340748 (34h14m11.704189501s) [upload=30026]: Done uploading segments...
Oct 31 21:54:40.008374502 (34h14m11.704223246s) [upload=30026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 6.307801796s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [X....................................X......................] duration:
5.938776567s trace:
https://jaeger.dev.storj.io/trace/16457411f40d1a7a
FAIL (2025-10-31 22:11:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.938838246s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..X...................................X.....................] duration:
3.911872621s trace:
https://jaeger.dev.storj.io/trace/74d5301c9b50695d
FAIL (2025-10-31 22:11:22 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.911942452s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [X.X................................XX.......................] duration:
5.889794318s trace:
https://jaeger.dev.storj.io/trace/777326af11170570
FAIL (2025-10-31 22:11:01 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 31 21:54:10.455403616 (34h13m42.151252369s) [upload=30026]: Starting upload
Oct 31 21:54:10.455522532 (34h13m42.151371285s) [upload=30026]: Done starting upload
Oct 31 21:54:10.455607515 (34h13m42.151456268s) [upload=30026]: Uploading segments...
Oct 31 21:54:10.455903067 (34h13m42.15175182s) [upload=30026]: Got next segment. Inline: false
Oct 31 21:54:10.455945959 (34h13m42.151794702s) [upload=30026]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:10.455953403 (34h13m42.151802146s) [upload=30026]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:10.703207265 (34h13m42.399056029s) [seg_pos={0 0}, upload=30026]: Begin upload segment...
Oct 31 21:54:10.703270286 (34h13m42.399119039s) [seg_pos={0 0}, upload=30026]: Done begin upload segment.
Oct 31 21:54:10.703295072 (34h13m42.399143825s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.703311003 (34h13m42.399159756s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.703452613 (34h13m42.399301366s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.704249731 (34h13m42.400098484s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.704696551 (34h13m42.400545305s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.704999748 (34h13m42.400848531s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.705356077 (34h13m42.40120483s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.705741331 (34h13m42.401590074s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.706164356 (34h13m42.402013109s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.706471972 (34h13m42.402320715s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.706819254 (34h13m42.402667997s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.707176183 (34h13m42.403024936s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.707611081 (34h13m42.403459834s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.708027884 (34h13m42.403876637s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.708460038 (34h13m42.404308781s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.708841935 (34h13m42.404690688s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:12.455521019 (34h13m44.151369772s) [upload=30026]: Next returned nil segment
Oct 31 21:54:12.455539354 (34h13m44.151388107s) [upload=30026]: Waiting for error group managing segments...
Oct 31 21:54:12.740273646 (34h13m44.436122389s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.742133407 (34h13m44.43798216s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.742843700 (34h13m44.438692443s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.743356016 (34h13m44.439204769s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.747176791 (34h13m44.443025535s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.749287941 (34h13m44.445136684s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.751847415 (34h13m44.447696168s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.754964321 (34h13m44.450813084s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.760193528 (34h13m44.456042281s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.763852014 (34h13m44.459700757s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.804034039 (34h13m44.499882792s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:39.766396977 (34h14m11.46224573s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection
Oct 31 21:54:39.766509131 (34h14m11.462357874s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:45860->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:39.766523238 (34h14m11.462371981s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 188.166.225.170:49310->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:39.766612227 (34h14m11.46246097s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:52980->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:39.766670488 (34h14m11.462519241s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 188.166.225.170:49096->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:39.766695265 (34h14m11.462544008s) [seg_pos={0 0}, upload=30026]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 31 21:54:40.008340748 (34h14m11.704189501s) [upload=30026]: Done uploading segments...
Oct 31 21:54:40.008374502 (34h14m11.704223246s) [upload=30026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.889828444s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.X....................................X.....................] duration:
5.75786737s trace:
https://jaeger.dev.storj.io/trace/16457411f40d1a7a
FAIL (2025-10-31 22:10:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.757936161s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [...X...................................X....................] duration:
3.583865016s trace:
https://jaeger.dev.storj.io/trace/74d5301c9b50695d
FAIL (2025-10-31 22:10:54 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.583900914s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.XX.................................X.......................] duration:
5.701338581s trace:
https://jaeger.dev.storj.io/trace/4a436e8f0e3652a9
FAIL (2025-10-31 22:10:49 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
=QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.841512922 (34h13m43.537361665s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.841695811 (34h13m43.537544554s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.842121341 (34h13m43.537970084s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.842258863 (34h13m43.538107606s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.842653173 (34h13m43.538501917s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.843123740 (34h13m43.538972493s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.843233278 (34h13m43.539082031s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.843685530 (34h13m43.539534283s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.844090440 (34h13m43.539939193s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.844193336 (34h13m43.540042089s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.844586345 (34h13m43.540435098s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.844901665 (34h13m43.540750418s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.845319241 (34h13m43.541167994s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.845683333 (34h13m43.541532086s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.845754208 (34h13m43.541602961s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.846188546 (34h13m43.542037289s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:13.235177054 (34h13m44.931025807s) [upload=30028]: Next returned nil segment
Oct 31 21:54:13.235196149 (34h13m44.931044902s) [upload=30028]: Waiting for error group managing segments...
Oct 31 21:54:13.731846523 (34h13m45.427695276s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.746649450 (34h13m45.442498203s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.749172355 (34h13m45.445021108s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.824326073 (34h13m45.520174826s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.838257411 (34h13m45.534106154s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.846435784 (34h13m45.542284527s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:14.712603264 (34h13m46.408452007s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:45.241231995 (34h14m16.937080748s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:47.222044337 (34h14m18.91789309s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:57.928194315 (34h14m29.624043068s) [noise=true , address=77.90.31.15:4719 , piece_id=QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:55:07.960970275 (34h14m39.656819028s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:55:10.903651570 (34h14m42.599500323s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1YZmFUbHJ3DfhJ1fCqMcy2tdDvmdPogiNAiFwduvcDgsw1QAZQ, address:45.91.116.10:4106): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:55:10.903817216 (34h14m42.599665969s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection
Oct 31 21:55:10.903935131 (34h14m42.599783884s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection
Oct 31 21:55:10.904041113 (34h14m42.599889866s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection
Oct 31 21:55:10.904246463 (34h14m42.600095216s) [noise=true , address=77.90.31.14:4674 , piece_id=62IZARJIWZYWAKAVHOP32MZUSTE35VBMA46K472HY44SFP63G5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, address:77.90.31.14:4674): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:55:10.904369097 (34h14m42.60021785s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection
Oct 31 21:55:10.904515886 (34h14m42.600364639s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:55:10.904544992 (34h14m42.600393735s) [seg_pos={0 0}, upload=30028]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Oct 31 21:55:11.152804097 (34h14m42.84865284s) [upload=30028]: Done uploading segments...
Oct 31 21:55:11.152854594 (34h14m42.848703337s) [upload=30028]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 5.701368168s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [XX.................................X........................] duration:
6.149399725s trace:
https://jaeger.dev.storj.io/trace/162122918c6518ce
FAIL (2025-10-31 22:10:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.149424552s
edge
gateway
us-select-1
|
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)antiblip:
1 out of 15 is failed (threshold=6) [..............X] duration:
627.197525ms remote:
"tcp:[2620:cb:2000::1]:443" target:
https://www.storj.io/ trace:
https://jaeger.dev.storj.io/trace/1ed0d71156ecf368
FAIL (2025-10-31 22:10:38 UTC)
orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:49584->[2620:cb:2000::1]:443: read: connection reset by peer
Duration: 627.214257ms
core
website
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.XX.................................X.......................] duration:
6.25215987s trace:
https://jaeger.dev.storj.io/trace/162122918c6518ce
FAIL (2025-10-31 22:10:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.252186231s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [XX.X................................XX......................] duration:
6.034939131s trace:
https://jaeger.dev.storj.io/trace/777326af11170570
FAIL (2025-10-31 22:10:29 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 31 21:54:10.455403616 (34h13m42.151252369s) [upload=30026]: Starting upload
Oct 31 21:54:10.455522532 (34h13m42.151371285s) [upload=30026]: Done starting upload
Oct 31 21:54:10.455607515 (34h13m42.151456268s) [upload=30026]: Uploading segments...
Oct 31 21:54:10.455903067 (34h13m42.15175182s) [upload=30026]: Got next segment. Inline: false
Oct 31 21:54:10.455945959 (34h13m42.151794702s) [upload=30026]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:10.455953403 (34h13m42.151802146s) [upload=30026]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:10.703207265 (34h13m42.399056029s) [seg_pos={0 0}, upload=30026]: Begin upload segment...
Oct 31 21:54:10.703270286 (34h13m42.399119039s) [seg_pos={0 0}, upload=30026]: Done begin upload segment.
Oct 31 21:54:10.703295072 (34h13m42.399143825s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.703311003 (34h13m42.399159756s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.703452613 (34h13m42.399301366s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.704249731 (34h13m42.400098484s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.704696551 (34h13m42.400545305s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.704999748 (34h13m42.400848531s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.705356077 (34h13m42.40120483s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.705741331 (34h13m42.401590074s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.706164356 (34h13m42.402013109s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.706471972 (34h13m42.402320715s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.706819254 (34h13m42.402667997s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.707176183 (34h13m42.403024936s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.707611081 (34h13m42.403459834s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.708027884 (34h13m42.403876637s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.708460038 (34h13m42.404308781s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.708841935 (34h13m42.404690688s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:12.455521019 (34h13m44.151369772s) [upload=30026]: Next returned nil segment
Oct 31 21:54:12.455539354 (34h13m44.151388107s) [upload=30026]: Waiting for error group managing segments...
Oct 31 21:54:12.740273646 (34h13m44.436122389s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.742133407 (34h13m44.43798216s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.742843700 (34h13m44.438692443s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.743356016 (34h13m44.439204769s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.747176791 (34h13m44.443025535s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.749287941 (34h13m44.445136684s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.751847415 (34h13m44.447696168s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.754964321 (34h13m44.450813084s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.760193528 (34h13m44.456042281s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.763852014 (34h13m44.459700757s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.804034039 (34h13m44.499882792s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:39.766396977 (34h14m11.46224573s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection
Oct 31 21:54:39.766509131 (34h14m11.462357874s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:45860->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:39.766523238 (34h14m11.462371981s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 188.166.225.170:49310->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:39.766612227 (34h14m11.46246097s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:52980->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:39.766670488 (34h14m11.462519241s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 188.166.225.170:49096->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:39.766695265 (34h14m11.462544008s) [seg_pos={0 0}, upload=30026]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 31 21:54:40.008340748 (34h14m11.704189501s) [upload=30026]: Done uploading segments...
Oct 31 21:54:40.008374502 (34h14m11.704223246s) [upload=30026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 6.034955312s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [..XX.................................X......................] duration:
5.666710416s trace:
https://jaeger.dev.storj.io/trace/4a436e8f0e3652a9
FAIL (2025-10-31 22:10:25 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
=QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.841512922 (34h13m43.537361665s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.841695811 (34h13m43.537544554s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.842121341 (34h13m43.537970084s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.842258863 (34h13m43.538107606s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.842653173 (34h13m43.538501917s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.843123740 (34h13m43.538972493s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.843233278 (34h13m43.539082031s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.843685530 (34h13m43.539534283s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.844090440 (34h13m43.539939193s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.844193336 (34h13m43.540042089s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.844586345 (34h13m43.540435098s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.844901665 (34h13m43.540750418s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.845319241 (34h13m43.541167994s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.845683333 (34h13m43.541532086s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.845754208 (34h13m43.541602961s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.846188546 (34h13m43.542037289s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:13.235177054 (34h13m44.931025807s) [upload=30028]: Next returned nil segment
Oct 31 21:54:13.235196149 (34h13m44.931044902s) [upload=30028]: Waiting for error group managing segments...
Oct 31 21:54:13.731846523 (34h13m45.427695276s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.746649450 (34h13m45.442498203s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.749172355 (34h13m45.445021108s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.824326073 (34h13m45.520174826s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.838257411 (34h13m45.534106154s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.846435784 (34h13m45.542284527s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:14.712603264 (34h13m46.408452007s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:45.241231995 (34h14m16.937080748s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:47.222044337 (34h14m18.91789309s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:57.928194315 (34h14m29.624043068s) [noise=true , address=77.90.31.15:4719 , piece_id=QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:55:07.960970275 (34h14m39.656819028s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:55:10.903651570 (34h14m42.599500323s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1YZmFUbHJ3DfhJ1fCqMcy2tdDvmdPogiNAiFwduvcDgsw1QAZQ, address:45.91.116.10:4106): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:55:10.903817216 (34h14m42.599665969s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection
Oct 31 21:55:10.903935131 (34h14m42.599783884s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection
Oct 31 21:55:10.904041113 (34h14m42.599889866s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection
Oct 31 21:55:10.904246463 (34h14m42.600095216s) [noise=true , address=77.90.31.14:4674 , piece_id=62IZARJIWZYWAKAVHOP32MZUSTE35VBMA46K472HY44SFP63G5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, address:77.90.31.14:4674): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:55:10.904369097 (34h14m42.60021785s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection
Oct 31 21:55:10.904515886 (34h14m42.600364639s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:55:10.904544992 (34h14m42.600393735s) [seg_pos={0 0}, upload=30028]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Oct 31 21:55:11.152804097 (34h14m42.84865284s) [upload=30028]: Done uploading segments...
Oct 31 21:55:11.152854594 (34h14m42.848703337s) [upload=30028]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 5.666736626s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..X....................................X....................] duration:
5.8176462s trace:
https://jaeger.dev.storj.io/trace/16457411f40d1a7a
FAIL (2025-10-31 22:10:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.817689513s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [....X...................................X...................] duration:
3.360249451s trace:
https://jaeger.dev.storj.io/trace/74d5301c9b50695d
FAIL (2025-10-31 22:10:00 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.360333471s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [...XX.................................X.....................] duration:
5.778538571s trace:
https://jaeger.dev.storj.io/trace/4a436e8f0e3652a9
FAIL (2025-10-31 22:09:55 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
=QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.841512922 (34h13m43.537361665s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.841695811 (34h13m43.537544554s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.842121341 (34h13m43.537970084s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.842258863 (34h13m43.538107606s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.842653173 (34h13m43.538501917s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.843123740 (34h13m43.538972493s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.843233278 (34h13m43.539082031s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.843685530 (34h13m43.539534283s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.844090440 (34h13m43.539939193s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.844193336 (34h13m43.540042089s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.844586345 (34h13m43.540435098s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.844901665 (34h13m43.540750418s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.845319241 (34h13m43.541167994s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.845683333 (34h13m43.541532086s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.845754208 (34h13m43.541602961s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.846188546 (34h13m43.542037289s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:13.235177054 (34h13m44.931025807s) [upload=30028]: Next returned nil segment
Oct 31 21:54:13.235196149 (34h13m44.931044902s) [upload=30028]: Waiting for error group managing segments...
Oct 31 21:54:13.731846523 (34h13m45.427695276s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.746649450 (34h13m45.442498203s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.749172355 (34h13m45.445021108s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.824326073 (34h13m45.520174826s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.838257411 (34h13m45.534106154s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.846435784 (34h13m45.542284527s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:14.712603264 (34h13m46.408452007s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:45.241231995 (34h14m16.937080748s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:47.222044337 (34h14m18.91789309s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:57.928194315 (34h14m29.624043068s) [noise=true , address=77.90.31.15:4719 , piece_id=QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:55:07.960970275 (34h14m39.656819028s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:55:10.903651570 (34h14m42.599500323s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1YZmFUbHJ3DfhJ1fCqMcy2tdDvmdPogiNAiFwduvcDgsw1QAZQ, address:45.91.116.10:4106): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:55:10.903817216 (34h14m42.599665969s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection
Oct 31 21:55:10.903935131 (34h14m42.599783884s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection
Oct 31 21:55:10.904041113 (34h14m42.599889866s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection
Oct 31 21:55:10.904246463 (34h14m42.600095216s) [noise=true , address=77.90.31.14:4674 , piece_id=62IZARJIWZYWAKAVHOP32MZUSTE35VBMA46K472HY44SFP63G5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, address:77.90.31.14:4674): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:55:10.904369097 (34h14m42.60021785s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection
Oct 31 21:55:10.904515886 (34h14m42.600364639s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:55:10.904544992 (34h14m42.600393735s) [seg_pos={0 0}, upload=30028]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Oct 31 21:55:11.152804097 (34h14m42.84865284s) [upload=30028]: Done uploading segments...
Oct 31 21:55:11.152854594 (34h14m42.848703337s) [upload=30028]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 5.778596161s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..XX.................................X......................] duration:
6.127641426s trace:
https://jaeger.dev.storj.io/trace/162122918c6518ce
FAIL (2025-10-31 22:09:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.127676223s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.XX.X................................XX.....................] duration:
5.922203604s trace:
https://jaeger.dev.storj.io/trace/777326af11170570
FAIL (2025-10-31 22:09:45 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 31 21:54:10.455403616 (34h13m42.151252369s) [upload=30026]: Starting upload
Oct 31 21:54:10.455522532 (34h13m42.151371285s) [upload=30026]: Done starting upload
Oct 31 21:54:10.455607515 (34h13m42.151456268s) [upload=30026]: Uploading segments...
Oct 31 21:54:10.455903067 (34h13m42.15175182s) [upload=30026]: Got next segment. Inline: false
Oct 31 21:54:10.455945959 (34h13m42.151794702s) [upload=30026]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:10.455953403 (34h13m42.151802146s) [upload=30026]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:10.703207265 (34h13m42.399056029s) [seg_pos={0 0}, upload=30026]: Begin upload segment...
Oct 31 21:54:10.703270286 (34h13m42.399119039s) [seg_pos={0 0}, upload=30026]: Done begin upload segment.
Oct 31 21:54:10.703295072 (34h13m42.399143825s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.703311003 (34h13m42.399159756s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.703452613 (34h13m42.399301366s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.704249731 (34h13m42.400098484s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.704696551 (34h13m42.400545305s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.704999748 (34h13m42.400848531s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.705356077 (34h13m42.40120483s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.705741331 (34h13m42.401590074s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.706164356 (34h13m42.402013109s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.706471972 (34h13m42.402320715s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.706819254 (34h13m42.402667997s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.707176183 (34h13m42.403024936s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.707611081 (34h13m42.403459834s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.708027884 (34h13m42.403876637s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.708460038 (34h13m42.404308781s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.708841935 (34h13m42.404690688s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:12.455521019 (34h13m44.151369772s) [upload=30026]: Next returned nil segment
Oct 31 21:54:12.455539354 (34h13m44.151388107s) [upload=30026]: Waiting for error group managing segments...
Oct 31 21:54:12.740273646 (34h13m44.436122389s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.742133407 (34h13m44.43798216s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.742843700 (34h13m44.438692443s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.743356016 (34h13m44.439204769s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.747176791 (34h13m44.443025535s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.749287941 (34h13m44.445136684s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.751847415 (34h13m44.447696168s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.754964321 (34h13m44.450813084s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.760193528 (34h13m44.456042281s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.763852014 (34h13m44.459700757s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.804034039 (34h13m44.499882792s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:39.766396977 (34h14m11.46224573s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection
Oct 31 21:54:39.766509131 (34h14m11.462357874s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:45860->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:39.766523238 (34h14m11.462371981s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 188.166.225.170:49310->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:39.766612227 (34h14m11.46246097s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:52980->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:39.766670488 (34h14m11.462519241s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 188.166.225.170:49096->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:39.766695265 (34h14m11.462544008s) [seg_pos={0 0}, upload=30026]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 31 21:54:40.008340748 (34h14m11.704189501s) [upload=30026]: Done uploading segments...
Oct 31 21:54:40.008374502 (34h14m11.704223246s) [upload=30026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.922238109s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...X....................................X...................] duration:
5.641442461s trace:
https://jaeger.dev.storj.io/trace/16457411f40d1a7a
FAIL (2025-10-31 22:09:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.641473199s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [....XX.................................X....................] duration:
5.675288398s trace:
https://jaeger.dev.storj.io/trace/4a436e8f0e3652a9
FAIL (2025-10-31 22:09:26 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
=QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.841512922 (34h13m43.537361665s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.841695811 (34h13m43.537544554s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.842121341 (34h13m43.537970084s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.842258863 (34h13m43.538107606s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.842653173 (34h13m43.538501917s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.843123740 (34h13m43.538972493s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.843233278 (34h13m43.539082031s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.843685530 (34h13m43.539534283s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.844090440 (34h13m43.539939193s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.844193336 (34h13m43.540042089s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.844586345 (34h13m43.540435098s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.844901665 (34h13m43.540750418s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.845319241 (34h13m43.541167994s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.845683333 (34h13m43.541532086s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.845754208 (34h13m43.541602961s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.846188546 (34h13m43.542037289s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:13.235177054 (34h13m44.931025807s) [upload=30028]: Next returned nil segment
Oct 31 21:54:13.235196149 (34h13m44.931044902s) [upload=30028]: Waiting for error group managing segments...
Oct 31 21:54:13.731846523 (34h13m45.427695276s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.746649450 (34h13m45.442498203s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.749172355 (34h13m45.445021108s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.824326073 (34h13m45.520174826s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.838257411 (34h13m45.534106154s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.846435784 (34h13m45.542284527s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:14.712603264 (34h13m46.408452007s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:45.241231995 (34h14m16.937080748s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:47.222044337 (34h14m18.91789309s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:57.928194315 (34h14m29.624043068s) [noise=true , address=77.90.31.15:4719 , piece_id=QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:55:07.960970275 (34h14m39.656819028s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:55:10.903651570 (34h14m42.599500323s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1YZmFUbHJ3DfhJ1fCqMcy2tdDvmdPogiNAiFwduvcDgsw1QAZQ, address:45.91.116.10:4106): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:55:10.903817216 (34h14m42.599665969s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection
Oct 31 21:55:10.903935131 (34h14m42.599783884s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection
Oct 31 21:55:10.904041113 (34h14m42.599889866s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection
Oct 31 21:55:10.904246463 (34h14m42.600095216s) [noise=true , address=77.90.31.14:4674 , piece_id=62IZARJIWZYWAKAVHOP32MZUSTE35VBMA46K472HY44SFP63G5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, address:77.90.31.14:4674): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:55:10.904369097 (34h14m42.60021785s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection
Oct 31 21:55:10.904515886 (34h14m42.600364639s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:55:10.904544992 (34h14m42.600393735s) [seg_pos={0 0}, upload=30028]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Oct 31 21:55:11.152804097 (34h14m42.84865284s) [upload=30028]: Done uploading segments...
Oct 31 21:55:11.152854594 (34h14m42.848703337s) [upload=30028]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 5.675306653s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.....X...................................X..................] duration:
3.39262107s trace:
https://jaeger.dev.storj.io/trace/74d5301c9b50695d
FAIL (2025-10-31 22:09:19 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.392649885s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...XX.................................X.....................] duration:
6.048773961s trace:
https://jaeger.dev.storj.io/trace/162122918c6518ce
FAIL (2025-10-31 22:09:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.048803327s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....XX.................................X...................] duration:
5.747103693s trace:
https://jaeger.dev.storj.io/trace/4a436e8f0e3652a9
FAIL (2025-10-31 22:08:48 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
=QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.841512922 (34h13m43.537361665s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.841695811 (34h13m43.537544554s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.842121341 (34h13m43.537970084s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.842258863 (34h13m43.538107606s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.842653173 (34h13m43.538501917s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.843123740 (34h13m43.538972493s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.843233278 (34h13m43.539082031s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.843685530 (34h13m43.539534283s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.844090440 (34h13m43.539939193s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.844193336 (34h13m43.540042089s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.844586345 (34h13m43.540435098s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.844901665 (34h13m43.540750418s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.845319241 (34h13m43.541167994s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.845683333 (34h13m43.541532086s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.845754208 (34h13m43.541602961s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.846188546 (34h13m43.542037289s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:13.235177054 (34h13m44.931025807s) [upload=30028]: Next returned nil segment
Oct 31 21:54:13.235196149 (34h13m44.931044902s) [upload=30028]: Waiting for error group managing segments...
Oct 31 21:54:13.731846523 (34h13m45.427695276s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.746649450 (34h13m45.442498203s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.749172355 (34h13m45.445021108s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.824326073 (34h13m45.520174826s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.838257411 (34h13m45.534106154s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.846435784 (34h13m45.542284527s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:14.712603264 (34h13m46.408452007s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:45.241231995 (34h14m16.937080748s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:47.222044337 (34h14m18.91789309s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:57.928194315 (34h14m29.624043068s) [noise=true , address=77.90.31.15:4719 , piece_id=QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:55:07.960970275 (34h14m39.656819028s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:55:10.903651570 (34h14m42.599500323s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1YZmFUbHJ3DfhJ1fCqMcy2tdDvmdPogiNAiFwduvcDgsw1QAZQ, address:45.91.116.10:4106): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:55:10.903817216 (34h14m42.599665969s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection
Oct 31 21:55:10.903935131 (34h14m42.599783884s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection
Oct 31 21:55:10.904041113 (34h14m42.599889866s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection
Oct 31 21:55:10.904246463 (34h14m42.600095216s) [noise=true , address=77.90.31.14:4674 , piece_id=62IZARJIWZYWAKAVHOP32MZUSTE35VBMA46K472HY44SFP63G5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, address:77.90.31.14:4674): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:55:10.904369097 (34h14m42.60021785s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection
Oct 31 21:55:10.904515886 (34h14m42.600364639s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:55:10.904544992 (34h14m42.600393735s) [seg_pos={0 0}, upload=30028]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Oct 31 21:55:11.152804097 (34h14m42.84865284s) [upload=30028]: Done uploading segments...
Oct 31 21:55:11.152854594 (34h14m42.848703337s) [upload=30028]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 5.74713394s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..XX.X................................XX....................] duration:
5.909730938s trace:
https://jaeger.dev.storj.io/trace/777326af11170570
FAIL (2025-10-31 22:08:41 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 31 21:54:10.455403616 (34h13m42.151252369s) [upload=30026]: Starting upload
Oct 31 21:54:10.455522532 (34h13m42.151371285s) [upload=30026]: Done starting upload
Oct 31 21:54:10.455607515 (34h13m42.151456268s) [upload=30026]: Uploading segments...
Oct 31 21:54:10.455903067 (34h13m42.15175182s) [upload=30026]: Got next segment. Inline: false
Oct 31 21:54:10.455945959 (34h13m42.151794702s) [upload=30026]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:10.455953403 (34h13m42.151802146s) [upload=30026]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:10.703207265 (34h13m42.399056029s) [seg_pos={0 0}, upload=30026]: Begin upload segment...
Oct 31 21:54:10.703270286 (34h13m42.399119039s) [seg_pos={0 0}, upload=30026]: Done begin upload segment.
Oct 31 21:54:10.703295072 (34h13m42.399143825s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.703311003 (34h13m42.399159756s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.703452613 (34h13m42.399301366s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.704249731 (34h13m42.400098484s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.704696551 (34h13m42.400545305s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.704999748 (34h13m42.400848531s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.705356077 (34h13m42.40120483s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.705741331 (34h13m42.401590074s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.706164356 (34h13m42.402013109s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.706471972 (34h13m42.402320715s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.706819254 (34h13m42.402667997s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.707176183 (34h13m42.403024936s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.707611081 (34h13m42.403459834s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.708027884 (34h13m42.403876637s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.708460038 (34h13m42.404308781s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.708841935 (34h13m42.404690688s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:12.455521019 (34h13m44.151369772s) [upload=30026]: Next returned nil segment
Oct 31 21:54:12.455539354 (34h13m44.151388107s) [upload=30026]: Waiting for error group managing segments...
Oct 31 21:54:12.740273646 (34h13m44.436122389s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.742133407 (34h13m44.43798216s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.742843700 (34h13m44.438692443s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.743356016 (34h13m44.439204769s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.747176791 (34h13m44.443025535s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.749287941 (34h13m44.445136684s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.751847415 (34h13m44.447696168s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.754964321 (34h13m44.450813084s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.760193528 (34h13m44.456042281s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.763852014 (34h13m44.459700757s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.804034039 (34h13m44.499882792s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:39.766396977 (34h14m11.46224573s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection
Oct 31 21:54:39.766509131 (34h14m11.462357874s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:45860->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:39.766523238 (34h14m11.462371981s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 188.166.225.170:49310->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:39.766612227 (34h14m11.46246097s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:52980->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:39.766670488 (34h14m11.462519241s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 188.166.225.170:49096->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:39.766695265 (34h14m11.462544008s) [seg_pos={0 0}, upload=30026]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 31 21:54:40.008340748 (34h14m11.704189501s) [upload=30026]: Done uploading segments...
Oct 31 21:54:40.008374502 (34h14m11.704223246s) [upload=30026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.909760456s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [......X...................................X.................] duration:
3.365424143s trace:
https://jaeger.dev.storj.io/trace/74d5301c9b50695d
FAIL (2025-10-31 22:08:39 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.365457097s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [....XX.................................X....................] duration:
6.003992649s trace:
https://jaeger.dev.storj.io/trace/162122918c6518ce
FAIL (2025-10-31 22:08:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.004016112s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [....X....................................X..................] duration:
5.687374822s trace:
https://jaeger.dev.storj.io/trace/16457411f40d1a7a
FAIL (2025-10-31 22:08:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.687394479s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...XX.X................................XX...................] duration:
5.829016308s trace:
https://jaeger.dev.storj.io/trace/777326af11170570
FAIL (2025-10-31 22:08:11 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 31 21:54:10.455403616 (34h13m42.151252369s) [upload=30026]: Starting upload
Oct 31 21:54:10.455522532 (34h13m42.151371285s) [upload=30026]: Done starting upload
Oct 31 21:54:10.455607515 (34h13m42.151456268s) [upload=30026]: Uploading segments...
Oct 31 21:54:10.455903067 (34h13m42.15175182s) [upload=30026]: Got next segment. Inline: false
Oct 31 21:54:10.455945959 (34h13m42.151794702s) [upload=30026]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:10.455953403 (34h13m42.151802146s) [upload=30026]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:10.703207265 (34h13m42.399056029s) [seg_pos={0 0}, upload=30026]: Begin upload segment...
Oct 31 21:54:10.703270286 (34h13m42.399119039s) [seg_pos={0 0}, upload=30026]: Done begin upload segment.
Oct 31 21:54:10.703295072 (34h13m42.399143825s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.703311003 (34h13m42.399159756s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.703452613 (34h13m42.399301366s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.704249731 (34h13m42.400098484s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.704696551 (34h13m42.400545305s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.704999748 (34h13m42.400848531s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.705356077 (34h13m42.40120483s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.705741331 (34h13m42.401590074s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.706164356 (34h13m42.402013109s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.706471972 (34h13m42.402320715s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.706819254 (34h13m42.402667997s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.707176183 (34h13m42.403024936s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.707611081 (34h13m42.403459834s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.708027884 (34h13m42.403876637s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.708460038 (34h13m42.404308781s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.708841935 (34h13m42.404690688s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:12.455521019 (34h13m44.151369772s) [upload=30026]: Next returned nil segment
Oct 31 21:54:12.455539354 (34h13m44.151388107s) [upload=30026]: Waiting for error group managing segments...
Oct 31 21:54:12.740273646 (34h13m44.436122389s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.742133407 (34h13m44.43798216s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.742843700 (34h13m44.438692443s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.743356016 (34h13m44.439204769s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.747176791 (34h13m44.443025535s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.749287941 (34h13m44.445136684s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.751847415 (34h13m44.447696168s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.754964321 (34h13m44.450813084s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.760193528 (34h13m44.456042281s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.763852014 (34h13m44.459700757s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.804034039 (34h13m44.499882792s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:39.766396977 (34h14m11.46224573s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection
Oct 31 21:54:39.766509131 (34h14m11.462357874s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:45860->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:39.766523238 (34h14m11.462371981s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 188.166.225.170:49310->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:39.766612227 (34h14m11.46246097s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:52980->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:39.766670488 (34h14m11.462519241s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 188.166.225.170:49096->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:39.766695265 (34h14m11.462544008s) [seg_pos={0 0}, upload=30026]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 31 21:54:40.008340748 (34h14m11.704189501s) [upload=30026]: Done uploading segments...
Oct 31 21:54:40.008374502 (34h14m11.704223246s) [upload=30026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.829046455s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [......XX.................................X..................] duration:
5.726345833s trace:
https://jaeger.dev.storj.io/trace/4a436e8f0e3652a9
FAIL (2025-10-31 22:07:52 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
=QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.841512922 (34h13m43.537361665s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.841695811 (34h13m43.537544554s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.842121341 (34h13m43.537970084s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.842258863 (34h13m43.538107606s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.842653173 (34h13m43.538501917s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.843123740 (34h13m43.538972493s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.843233278 (34h13m43.539082031s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.843685530 (34h13m43.539534283s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.844090440 (34h13m43.539939193s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.844193336 (34h13m43.540042089s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.844586345 (34h13m43.540435098s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.844901665 (34h13m43.540750418s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.845319241 (34h13m43.541167994s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.845683333 (34h13m43.541532086s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.845754208 (34h13m43.541602961s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.846188546 (34h13m43.542037289s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:13.235177054 (34h13m44.931025807s) [upload=30028]: Next returned nil segment
Oct 31 21:54:13.235196149 (34h13m44.931044902s) [upload=30028]: Waiting for error group managing segments...
Oct 31 21:54:13.731846523 (34h13m45.427695276s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.746649450 (34h13m45.442498203s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.749172355 (34h13m45.445021108s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.824326073 (34h13m45.520174826s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.838257411 (34h13m45.534106154s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.846435784 (34h13m45.542284527s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:14.712603264 (34h13m46.408452007s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:45.241231995 (34h14m16.937080748s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:47.222044337 (34h14m18.91789309s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:57.928194315 (34h14m29.624043068s) [noise=true , address=77.90.31.15:4719 , piece_id=QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:55:07.960970275 (34h14m39.656819028s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:55:10.903651570 (34h14m42.599500323s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1YZmFUbHJ3DfhJ1fCqMcy2tdDvmdPogiNAiFwduvcDgsw1QAZQ, address:45.91.116.10:4106): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:55:10.903817216 (34h14m42.599665969s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection
Oct 31 21:55:10.903935131 (34h14m42.599783884s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection
Oct 31 21:55:10.904041113 (34h14m42.599889866s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection
Oct 31 21:55:10.904246463 (34h14m42.600095216s) [noise=true , address=77.90.31.14:4674 , piece_id=62IZARJIWZYWAKAVHOP32MZUSTE35VBMA46K472HY44SFP63G5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, address:77.90.31.14:4674): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:55:10.904369097 (34h14m42.60021785s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection
Oct 31 21:55:10.904515886 (34h14m42.600364639s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:55:10.904544992 (34h14m42.600393735s) [seg_pos={0 0}, upload=30028]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Oct 31 21:55:11.152804097 (34h14m42.84865284s) [upload=30028]: Done uploading segments...
Oct 31 21:55:11.152854594 (34h14m42.848703337s) [upload=30028]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 5.726372355s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [X......X...................................X................] duration:
3.451677658s trace:
https://jaeger.dev.storj.io/trace/74d5301c9b50695d
FAIL (2025-10-31 22:07:52 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.451704459s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....XX.................................X...................] duration:
6.128765886s trace:
https://jaeger.dev.storj.io/trace/162122918c6518ce
FAIL (2025-10-31 22:07:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.12879463s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [....XX.X................................XX..................] duration:
6.133553814s trace:
https://jaeger.dev.storj.io/trace/777326af11170570
FAIL (2025-10-31 22:07:37 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 31 21:54:10.455403616 (34h13m42.151252369s) [upload=30026]: Starting upload
Oct 31 21:54:10.455522532 (34h13m42.151371285s) [upload=30026]: Done starting upload
Oct 31 21:54:10.455607515 (34h13m42.151456268s) [upload=30026]: Uploading segments...
Oct 31 21:54:10.455903067 (34h13m42.15175182s) [upload=30026]: Got next segment. Inline: false
Oct 31 21:54:10.455945959 (34h13m42.151794702s) [upload=30026]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:10.455953403 (34h13m42.151802146s) [upload=30026]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:10.703207265 (34h13m42.399056029s) [seg_pos={0 0}, upload=30026]: Begin upload segment...
Oct 31 21:54:10.703270286 (34h13m42.399119039s) [seg_pos={0 0}, upload=30026]: Done begin upload segment.
Oct 31 21:54:10.703295072 (34h13m42.399143825s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.703311003 (34h13m42.399159756s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.703452613 (34h13m42.399301366s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.704249731 (34h13m42.400098484s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.704696551 (34h13m42.400545305s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.704999748 (34h13m42.400848531s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.705356077 (34h13m42.40120483s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.705741331 (34h13m42.401590074s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.706164356 (34h13m42.402013109s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.706471972 (34h13m42.402320715s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.706819254 (34h13m42.402667997s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.707176183 (34h13m42.403024936s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.707611081 (34h13m42.403459834s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.708027884 (34h13m42.403876637s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.708460038 (34h13m42.404308781s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.708841935 (34h13m42.404690688s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:12.455521019 (34h13m44.151369772s) [upload=30026]: Next returned nil segment
Oct 31 21:54:12.455539354 (34h13m44.151388107s) [upload=30026]: Waiting for error group managing segments...
Oct 31 21:54:12.740273646 (34h13m44.436122389s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.742133407 (34h13m44.43798216s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.742843700 (34h13m44.438692443s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.743356016 (34h13m44.439204769s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.747176791 (34h13m44.443025535s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.749287941 (34h13m44.445136684s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.751847415 (34h13m44.447696168s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.754964321 (34h13m44.450813084s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.760193528 (34h13m44.456042281s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.763852014 (34h13m44.459700757s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.804034039 (34h13m44.499882792s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:39.766396977 (34h14m11.46224573s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection
Oct 31 21:54:39.766509131 (34h14m11.462357874s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:45860->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:39.766523238 (34h14m11.462371981s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 188.166.225.170:49310->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:39.766612227 (34h14m11.46246097s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:52980->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:39.766670488 (34h14m11.462519241s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 188.166.225.170:49096->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:39.766695265 (34h14m11.462544008s) [seg_pos={0 0}, upload=30026]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 31 21:54:40.008340748 (34h14m11.704189501s) [upload=30026]: Done uploading segments...
Oct 31 21:54:40.008374502 (34h14m11.704223246s) [upload=30026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 6.133581908s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [X....X....................................X.................] duration:
5.665975225s trace:
https://jaeger.dev.storj.io/trace/16457411f40d1a7a
FAIL (2025-10-31 22:07:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.666003277s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [X......XX.................................X.................] duration:
5.870512526s trace:
https://jaeger.dev.storj.io/trace/4a436e8f0e3652a9
FAIL (2025-10-31 22:07:10 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
=QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.841512922 (34h13m43.537361665s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.841695811 (34h13m43.537544554s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.842121341 (34h13m43.537970084s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.842258863 (34h13m43.538107606s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.842653173 (34h13m43.538501917s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.843123740 (34h13m43.538972493s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.843233278 (34h13m43.539082031s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.843685530 (34h13m43.539534283s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.844090440 (34h13m43.539939193s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.844193336 (34h13m43.540042089s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.844586345 (34h13m43.540435098s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.844901665 (34h13m43.540750418s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.845319241 (34h13m43.541167994s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.845683333 (34h13m43.541532086s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.845754208 (34h13m43.541602961s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.846188546 (34h13m43.542037289s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:13.235177054 (34h13m44.931025807s) [upload=30028]: Next returned nil segment
Oct 31 21:54:13.235196149 (34h13m44.931044902s) [upload=30028]: Waiting for error group managing segments...
Oct 31 21:54:13.731846523 (34h13m45.427695276s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.746649450 (34h13m45.442498203s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.749172355 (34h13m45.445021108s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.824326073 (34h13m45.520174826s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.838257411 (34h13m45.534106154s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.846435784 (34h13m45.542284527s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:14.712603264 (34h13m46.408452007s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:45.241231995 (34h14m16.937080748s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:47.222044337 (34h14m18.91789309s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:57.928194315 (34h14m29.624043068s) [noise=true , address=77.90.31.15:4719 , piece_id=QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:55:07.960970275 (34h14m39.656819028s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:55:10.903651570 (34h14m42.599500323s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1YZmFUbHJ3DfhJ1fCqMcy2tdDvmdPogiNAiFwduvcDgsw1QAZQ, address:45.91.116.10:4106): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:55:10.903817216 (34h14m42.599665969s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection
Oct 31 21:55:10.903935131 (34h14m42.599783884s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection
Oct 31 21:55:10.904041113 (34h14m42.599889866s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection
Oct 31 21:55:10.904246463 (34h14m42.600095216s) [noise=true , address=77.90.31.14:4674 , piece_id=62IZARJIWZYWAKAVHOP32MZUSTE35VBMA46K472HY44SFP63G5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, address:77.90.31.14:4674): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:55:10.904369097 (34h14m42.60021785s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection
Oct 31 21:55:10.904515886 (34h14m42.600364639s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:55:10.904544992 (34h14m42.600393735s) [seg_pos={0 0}, upload=30028]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Oct 31 21:55:11.152804097 (34h14m42.84865284s) [upload=30028]: Done uploading segments...
Oct 31 21:55:11.152854594 (34h14m42.848703337s) [upload=30028]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 5.870533517s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.....XX.X................................XX.................] duration:
6.096561814s trace:
https://jaeger.dev.storj.io/trace/777326af11170570
FAIL (2025-10-31 22:07:02 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 31 21:54:10.455403616 (34h13m42.151252369s) [upload=30026]: Starting upload
Oct 31 21:54:10.455522532 (34h13m42.151371285s) [upload=30026]: Done starting upload
Oct 31 21:54:10.455607515 (34h13m42.151456268s) [upload=30026]: Uploading segments...
Oct 31 21:54:10.455903067 (34h13m42.15175182s) [upload=30026]: Got next segment. Inline: false
Oct 31 21:54:10.455945959 (34h13m42.151794702s) [upload=30026]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:10.455953403 (34h13m42.151802146s) [upload=30026]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:10.703207265 (34h13m42.399056029s) [seg_pos={0 0}, upload=30026]: Begin upload segment...
Oct 31 21:54:10.703270286 (34h13m42.399119039s) [seg_pos={0 0}, upload=30026]: Done begin upload segment.
Oct 31 21:54:10.703295072 (34h13m42.399143825s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.703311003 (34h13m42.399159756s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.703452613 (34h13m42.399301366s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.704249731 (34h13m42.400098484s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.704696551 (34h13m42.400545305s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.704999748 (34h13m42.400848531s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.705356077 (34h13m42.40120483s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.705741331 (34h13m42.401590074s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.706164356 (34h13m42.402013109s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.706471972 (34h13m42.402320715s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.706819254 (34h13m42.402667997s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.707176183 (34h13m42.403024936s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.707611081 (34h13m42.403459834s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.708027884 (34h13m42.403876637s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.708460038 (34h13m42.404308781s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.708841935 (34h13m42.404690688s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:12.455521019 (34h13m44.151369772s) [upload=30026]: Next returned nil segment
Oct 31 21:54:12.455539354 (34h13m44.151388107s) [upload=30026]: Waiting for error group managing segments...
Oct 31 21:54:12.740273646 (34h13m44.436122389s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.742133407 (34h13m44.43798216s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.742843700 (34h13m44.438692443s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.743356016 (34h13m44.439204769s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.747176791 (34h13m44.443025535s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.749287941 (34h13m44.445136684s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.751847415 (34h13m44.447696168s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.754964321 (34h13m44.450813084s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.760193528 (34h13m44.456042281s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.763852014 (34h13m44.459700757s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.804034039 (34h13m44.499882792s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:39.766396977 (34h14m11.46224573s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection
Oct 31 21:54:39.766509131 (34h14m11.462357874s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:45860->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:39.766523238 (34h14m11.462371981s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 188.166.225.170:49310->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:39.766612227 (34h14m11.46246097s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:52980->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:39.766670488 (34h14m11.462519241s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 188.166.225.170:49096->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:39.766695265 (34h14m11.462544008s) [seg_pos={0 0}, upload=30026]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 31 21:54:40.008340748 (34h14m11.704189501s) [upload=30026]: Done uploading segments...
Oct 31 21:54:40.008374502 (34h14m11.704223246s) [upload=30026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 6.096586591s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [......XX.................................X..................] duration:
6.123000456s trace:
https://jaeger.dev.storj.io/trace/162122918c6518ce
FAIL (2025-10-31 22:06:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.123031275s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [XX......X...................................X...............] duration:
3.475075454s trace:
https://jaeger.dev.storj.io/trace/74d5301c9b50695d
FAIL (2025-10-31 22:06:51 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.475093498s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [XX....X....................................X................] duration:
5.684333242s trace:
https://jaeger.dev.storj.io/trace/16457411f40d1a7a
FAIL (2025-10-31 22:06:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.684399867s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [X......XX.................................X.................] duration:
6.166725203s trace:
https://jaeger.dev.storj.io/trace/162122918c6518ce
FAIL (2025-10-31 22:06:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.166757945s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.XX....X....................................X...............] duration:
5.654363015s trace:
https://jaeger.dev.storj.io/trace/16457411f40d1a7a
FAIL (2025-10-31 22:06:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.654383373s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.X......XX.................................X................] duration:
5.878181323s trace:
https://jaeger.dev.storj.io/trace/4a436e8f0e3652a9
FAIL (2025-10-31 22:06:05 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
=QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.841512922 (34h13m43.537361665s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.841695811 (34h13m43.537544554s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.842121341 (34h13m43.537970084s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.842258863 (34h13m43.538107606s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.842653173 (34h13m43.538501917s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.843123740 (34h13m43.538972493s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.843233278 (34h13m43.539082031s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.843685530 (34h13m43.539534283s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.844090440 (34h13m43.539939193s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.844193336 (34h13m43.540042089s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.844586345 (34h13m43.540435098s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.844901665 (34h13m43.540750418s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.845319241 (34h13m43.541167994s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.845683333 (34h13m43.541532086s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.845754208 (34h13m43.541602961s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.846188546 (34h13m43.542037289s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:13.235177054 (34h13m44.931025807s) [upload=30028]: Next returned nil segment
Oct 31 21:54:13.235196149 (34h13m44.931044902s) [upload=30028]: Waiting for error group managing segments...
Oct 31 21:54:13.731846523 (34h13m45.427695276s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.746649450 (34h13m45.442498203s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.749172355 (34h13m45.445021108s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.824326073 (34h13m45.520174826s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.838257411 (34h13m45.534106154s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.846435784 (34h13m45.542284527s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:14.712603264 (34h13m46.408452007s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:45.241231995 (34h14m16.937080748s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:47.222044337 (34h14m18.91789309s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:57.928194315 (34h14m29.624043068s) [noise=true , address=77.90.31.15:4719 , piece_id=QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:55:07.960970275 (34h14m39.656819028s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:55:10.903651570 (34h14m42.599500323s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1YZmFUbHJ3DfhJ1fCqMcy2tdDvmdPogiNAiFwduvcDgsw1QAZQ, address:45.91.116.10:4106): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:55:10.903817216 (34h14m42.599665969s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection
Oct 31 21:55:10.903935131 (34h14m42.599783884s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection
Oct 31 21:55:10.904041113 (34h14m42.599889866s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection
Oct 31 21:55:10.904246463 (34h14m42.600095216s) [noise=true , address=77.90.31.14:4674 , piece_id=62IZARJIWZYWAKAVHOP32MZUSTE35VBMA46K472HY44SFP63G5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, address:77.90.31.14:4674): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:55:10.904369097 (34h14m42.60021785s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection
Oct 31 21:55:10.904515886 (34h14m42.600364639s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:55:10.904544992 (34h14m42.600393735s) [seg_pos={0 0}, upload=30028]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Oct 31 21:55:11.152804097 (34h14m42.84865284s) [upload=30028]: Done uploading segments...
Oct 31 21:55:11.152854594 (34h14m42.848703337s) [upload=30028]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 5.878244895s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.XX......X...................................X..............] duration:
3.430762578s trace:
https://jaeger.dev.storj.io/trace/74d5301c9b50695d
FAIL (2025-10-31 22:06:00 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.43081161s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [..XX....X....................................X..............] duration:
5.726490481s trace:
https://jaeger.dev.storj.io/trace/16457411f40d1a7a
FAIL (2025-10-31 22:05:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.726511682s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [......XX.X................................XX................] duration:
6.397862281s trace:
https://jaeger.dev.storj.io/trace/777326af11170570
FAIL (2025-10-31 22:05:48 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 31 21:54:10.455403616 (34h13m42.151252369s) [upload=30026]: Starting upload
Oct 31 21:54:10.455522532 (34h13m42.151371285s) [upload=30026]: Done starting upload
Oct 31 21:54:10.455607515 (34h13m42.151456268s) [upload=30026]: Uploading segments...
Oct 31 21:54:10.455903067 (34h13m42.15175182s) [upload=30026]: Got next segment. Inline: false
Oct 31 21:54:10.455945959 (34h13m42.151794702s) [upload=30026]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:10.455953403 (34h13m42.151802146s) [upload=30026]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:10.703207265 (34h13m42.399056029s) [seg_pos={0 0}, upload=30026]: Begin upload segment...
Oct 31 21:54:10.703270286 (34h13m42.399119039s) [seg_pos={0 0}, upload=30026]: Done begin upload segment.
Oct 31 21:54:10.703295072 (34h13m42.399143825s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.703311003 (34h13m42.399159756s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.703452613 (34h13m42.399301366s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.704249731 (34h13m42.400098484s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.704696551 (34h13m42.400545305s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.704999748 (34h13m42.400848531s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.705356077 (34h13m42.40120483s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.705741331 (34h13m42.401590074s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.706164356 (34h13m42.402013109s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.706471972 (34h13m42.402320715s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.706819254 (34h13m42.402667997s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.707176183 (34h13m42.403024936s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.707611081 (34h13m42.403459834s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.708027884 (34h13m42.403876637s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.708460038 (34h13m42.404308781s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.708841935 (34h13m42.404690688s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:12.455521019 (34h13m44.151369772s) [upload=30026]: Next returned nil segment
Oct 31 21:54:12.455539354 (34h13m44.151388107s) [upload=30026]: Waiting for error group managing segments...
Oct 31 21:54:12.740273646 (34h13m44.436122389s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.742133407 (34h13m44.43798216s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.742843700 (34h13m44.438692443s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.743356016 (34h13m44.439204769s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.747176791 (34h13m44.443025535s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.749287941 (34h13m44.445136684s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.751847415 (34h13m44.447696168s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.754964321 (34h13m44.450813084s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.760193528 (34h13m44.456042281s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.763852014 (34h13m44.459700757s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.804034039 (34h13m44.499882792s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:39.766396977 (34h14m11.46224573s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection
Oct 31 21:54:39.766509131 (34h14m11.462357874s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:45860->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:39.766523238 (34h14m11.462371981s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 188.166.225.170:49310->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:39.766612227 (34h14m11.46246097s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:52980->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:39.766670488 (34h14m11.462519241s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 188.166.225.170:49096->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:39.766695265 (34h14m11.462544008s) [seg_pos={0 0}, upload=30026]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 31 21:54:40.008340748 (34h14m11.704189501s) [upload=30026]: Done uploading segments...
Oct 31 21:54:40.008374502 (34h14m11.704223246s) [upload=30026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 6.397897367s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [..X......XX.................................X...............] duration:
5.721420259s trace:
https://jaeger.dev.storj.io/trace/4a436e8f0e3652a9
FAIL (2025-10-31 22:05:47 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
=QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.841512922 (34h13m43.537361665s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.841695811 (34h13m43.537544554s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.842121341 (34h13m43.537970084s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.842258863 (34h13m43.538107606s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.842653173 (34h13m43.538501917s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.843123740 (34h13m43.538972493s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.843233278 (34h13m43.539082031s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.843685530 (34h13m43.539534283s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.844090440 (34h13m43.539939193s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.844193336 (34h13m43.540042089s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.844586345 (34h13m43.540435098s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.844901665 (34h13m43.540750418s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.845319241 (34h13m43.541167994s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.845683333 (34h13m43.541532086s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.845754208 (34h13m43.541602961s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:11.846188546 (34h13m43.542037289s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Uploading piece...
Oct 31 21:54:13.235177054 (34h13m44.931025807s) [upload=30028]: Next returned nil segment
Oct 31 21:54:13.235196149 (34h13m44.931044902s) [upload=30028]: Waiting for error group managing segments...
Oct 31 21:54:13.731846523 (34h13m45.427695276s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.746649450 (34h13m45.442498203s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.749172355 (34h13m45.445021108s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.824326073 (34h13m45.520174826s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.838257411 (34h13m45.534106154s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:13.846435784 (34h13m45.542284527s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:14.712603264 (34h13m46.408452007s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:45.241231995 (34h14m16.937080748s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:47.222044337 (34h14m18.91789309s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:54:57.928194315 (34h14m29.624043068s) [noise=true , address=77.90.31.15:4719 , piece_id=QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:55:07.960970275 (34h14m39.656819028s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil>
Oct 31 21:55:10.903651570 (34h14m42.599500323s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1YZmFUbHJ3DfhJ1fCqMcy2tdDvmdPogiNAiFwduvcDgsw1QAZQ, address:45.91.116.10:4106): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:55:10.903817216 (34h14m42.599665969s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection
Oct 31 21:55:10.903935131 (34h14m42.599783884s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection
Oct 31 21:55:10.904041113 (34h14m42.599889866s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection
Oct 31 21:55:10.904246463 (34h14m42.600095216s) [noise=true , address=77.90.31.14:4674 , piece_id=62IZARJIWZYWAKAVHOP32MZUSTE35VBMA46K472HY44SFP63G5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, address:77.90.31.14:4674): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:55:10.904369097 (34h14m42.60021785s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection
Oct 31 21:55:10.904515886 (34h14m42.600364639s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:55:10.904544992 (34h14m42.600393735s) [seg_pos={0 0}, upload=30028]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Oct 31 21:55:11.152804097 (34h14m42.84865284s) [upload=30028]: Done uploading segments...
Oct 31 21:55:11.152854594 (34h14m42.848703337s) [upload=30028]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 5.721503097s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [XX......XX.................................X................] duration:
6.158734426s trace:
https://jaeger.dev.storj.io/trace/162122918c6518ce
FAIL (2025-10-31 22:05:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.158756447s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.......XX.X................................XX...............] duration:
6.103324178s trace:
https://jaeger.dev.storj.io/trace/777326af11170570
FAIL (2025-10-31 22:05:26 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 31 21:54:10.455403616 (34h13m42.151252369s) [upload=30026]: Starting upload
Oct 31 21:54:10.455522532 (34h13m42.151371285s) [upload=30026]: Done starting upload
Oct 31 21:54:10.455607515 (34h13m42.151456268s) [upload=30026]: Uploading segments...
Oct 31 21:54:10.455903067 (34h13m42.15175182s) [upload=30026]: Got next segment. Inline: false
Oct 31 21:54:10.455945959 (34h13m42.151794702s) [upload=30026]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:10.455953403 (34h13m42.151802146s) [upload=30026]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:10.703207265 (34h13m42.399056029s) [seg_pos={0 0}, upload=30026]: Begin upload segment...
Oct 31 21:54:10.703270286 (34h13m42.399119039s) [seg_pos={0 0}, upload=30026]: Done begin upload segment.
Oct 31 21:54:10.703295072 (34h13m42.399143825s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.703311003 (34h13m42.399159756s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.703452613 (34h13m42.399301366s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.704249731 (34h13m42.400098484s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.704696551 (34h13m42.400545305s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.704999748 (34h13m42.400848531s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.705356077 (34h13m42.40120483s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.705741331 (34h13m42.401590074s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.706164356 (34h13m42.402013109s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.706471972 (34h13m42.402320715s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.706819254 (34h13m42.402667997s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.707176183 (34h13m42.403024936s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.707611081 (34h13m42.403459834s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.708027884 (34h13m42.403876637s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.708460038 (34h13m42.404308781s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:10.708841935 (34h13m42.404690688s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Uploading piece...
Oct 31 21:54:12.455521019 (34h13m44.151369772s) [upload=30026]: Next returned nil segment
Oct 31 21:54:12.455539354 (34h13m44.151388107s) [upload=30026]: Waiting for error group managing segments...
Oct 31 21:54:12.740273646 (34h13m44.436122389s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.742133407 (34h13m44.43798216s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.742843700 (34h13m44.438692443s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.743356016 (34h13m44.439204769s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.747176791 (34h13m44.443025535s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.749287941 (34h13m44.445136684s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.751847415 (34h13m44.447696168s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.754964321 (34h13m44.450813084s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.760193528 (34h13m44.456042281s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.763852014 (34h13m44.459700757s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:12.804034039 (34h13m44.499882792s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil>
Oct 31 21:54:39.766396977 (34h14m11.46224573s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection
Oct 31 21:54:39.766509131 (34h14m11.462357874s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:45860->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:39.766523238 (34h14m11.462371981s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 188.166.225.170:49310->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:39.766612227 (34h14m11.46246097s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:52980->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:39.766670488 (34h14m11.462519241s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 188.166.225.170:49096->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:39.766695265 (34h14m11.462544008s) [seg_pos={0 0}, upload=30026]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 31 21:54:40.008340748 (34h14m11.704189501s) [upload=30026]: Done uploading segments...
Oct 31 21:54:40.008374502 (34h14m11.704223246s) [upload=30026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 6.103357392s
core
uplink
us-select-1
|