Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..............XX....X...................XX..................] duration: 3.422202293s trace: https://jaeger.dev.storj.io/trace/2d7e37a5218d47ae FAIL (2025-12-16 17:37:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.42222738s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................X......X......X......] duration: 5.767374479s trace: https://jaeger.dev.storj.io/trace/3a8b2eb64e78cdb7 FAIL (2025-12-16 17:37:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 17:33:25.825412431 (266h4m25.696495476s) [upload=233226]: Starting upload Dec 16 17:33:25.825716299 (266h4m25.696799334s) [upload=233226]: Done starting upload Dec 16 17:33:25.825900039 (266h4m25.696983074s) [upload=233226]: Uploading segments... Dec 16 17:33:25.825986874 (266h4m25.697069919s) [upload=233226]: Got next segment. Inline: false Dec 16 17:33:25.826024115 (266h4m25.69710716s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:33:25.826029405 (266h4m25.69711245s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:33:26.081528100 (266h4m25.952611145s) [seg_pos={0 0}, upload=233226]: Begin upload segment... Dec 16 17:33:26.081578937 (266h4m25.952661972s) [seg_pos={0 0}, upload=233226]: Done begin upload segment. Dec 16 17:33:26.081646916 (266h4m25.952729951s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.081689087 (266h4m25.952772122s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.082557751 (266h4m25.953640796s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083162883 (266h4m25.954245928s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083811628 (266h4m25.954894674s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.084268108 (266h4m25.955351153s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086136556 (266h4m25.957219591s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086764421 (266h4m25.957847456s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.087471016 (266h4m25.958554061s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.088841396 (266h4m25.959924442s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.089031668 (266h4m25.960114703s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.090373093 (266h4m25.961456128s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.092053242 (266h4m25.963136287s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.095858979 (266h4m25.966942024s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.096244182 (266h4m25.967327227s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.099898400 (266h4m25.970981445s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:27.793861758 (266h4m27.664944803s) [upload=233226]: Next returned nil segment Dec 16 17:33:27.793883430 (266h4m27.664966465s) [upload=233226]: Waiting for error group managing segments... Dec 16 17:33:28.076919891 (266h4m27.948002936s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.080532791 (266h4m27.951615826s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.081529298 (266h4m27.952612333s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.088220410 (266h4m27.959303455s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.102098585 (266h4m27.97318163s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.110557603 (266h4m27.981640648s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.112773352 (266h4m27.983856387s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.168421046 (266h4m28.039504091s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.175508041 (266h4m28.046591076s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.367477684 (266h4m28.23856073s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.388567654 (266h4m28.259650699s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.393640814 (266h4m28.264723859s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466522674 (266h4m28.337605719s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466547552 (266h4m28.337630597s) [seg_pos={0 0}, upload=233226]: Segment reached optimal threshold of 13 pieces. Dec 16 17:33:28.466920051 (266h4m28.338003096s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466932094 (266h4m28.338015129s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.466960198 (266h4m28.338043243s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466972481 (266h4m28.338055526s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467172281 (266h4m28.338255336s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection Dec 16 17:33:28.467182901 (266h4m28.338265946s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467216766 (266h4m28.338299811s) [seg_pos={0 0}, upload=233226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:33:28.467315314 (266h4m28.338398349s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:33:28.467322477 (266h4m28.338405522s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:33:28.799678512 (266h4m28.670761547s) [upload=233226]: Done uploading segments... Dec 16 17:33:28.799699542 (266h4m28.670782587s) [upload=233226]: Upload finished. err: <nil> Duration: 5.767402964s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 954.610248ms trace: https://jaeger.dev.storj.io/trace/2130d3e94d4b0821 FAIL (2025-12-16 17:36:42 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37318->45.38.24.2:443: read: connection reset by peer Duration: 954.635796ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [...............XX....X...................XX.................] duration: 3.734194533s trace: https://jaeger.dev.storj.io/trace/2d7e37a5218d47ae FAIL (2025-12-16 17:36:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.734225962s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................X......X......X.....] duration: 6.110764024s trace: https://jaeger.dev.storj.io/trace/3a8b2eb64e78cdb7 FAIL (2025-12-16 17:36:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 17:33:25.825412431 (266h4m25.696495476s) [upload=233226]: Starting upload Dec 16 17:33:25.825716299 (266h4m25.696799334s) [upload=233226]: Done starting upload Dec 16 17:33:25.825900039 (266h4m25.696983074s) [upload=233226]: Uploading segments... Dec 16 17:33:25.825986874 (266h4m25.697069919s) [upload=233226]: Got next segment. Inline: false Dec 16 17:33:25.826024115 (266h4m25.69710716s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:33:25.826029405 (266h4m25.69711245s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:33:26.081528100 (266h4m25.952611145s) [seg_pos={0 0}, upload=233226]: Begin upload segment... Dec 16 17:33:26.081578937 (266h4m25.952661972s) [seg_pos={0 0}, upload=233226]: Done begin upload segment. Dec 16 17:33:26.081646916 (266h4m25.952729951s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.081689087 (266h4m25.952772122s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.082557751 (266h4m25.953640796s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083162883 (266h4m25.954245928s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083811628 (266h4m25.954894674s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.084268108 (266h4m25.955351153s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086136556 (266h4m25.957219591s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086764421 (266h4m25.957847456s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.087471016 (266h4m25.958554061s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.088841396 (266h4m25.959924442s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.089031668 (266h4m25.960114703s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.090373093 (266h4m25.961456128s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.092053242 (266h4m25.963136287s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.095858979 (266h4m25.966942024s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.096244182 (266h4m25.967327227s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.099898400 (266h4m25.970981445s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:27.793861758 (266h4m27.664944803s) [upload=233226]: Next returned nil segment Dec 16 17:33:27.793883430 (266h4m27.664966465s) [upload=233226]: Waiting for error group managing segments... Dec 16 17:33:28.076919891 (266h4m27.948002936s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.080532791 (266h4m27.951615826s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.081529298 (266h4m27.952612333s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.088220410 (266h4m27.959303455s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.102098585 (266h4m27.97318163s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.110557603 (266h4m27.981640648s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.112773352 (266h4m27.983856387s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.168421046 (266h4m28.039504091s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.175508041 (266h4m28.046591076s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.367477684 (266h4m28.23856073s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.388567654 (266h4m28.259650699s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.393640814 (266h4m28.264723859s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466522674 (266h4m28.337605719s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466547552 (266h4m28.337630597s) [seg_pos={0 0}, upload=233226]: Segment reached optimal threshold of 13 pieces. Dec 16 17:33:28.466920051 (266h4m28.338003096s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466932094 (266h4m28.338015129s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.466960198 (266h4m28.338043243s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466972481 (266h4m28.338055526s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467172281 (266h4m28.338255336s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection Dec 16 17:33:28.467182901 (266h4m28.338265946s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467216766 (266h4m28.338299811s) [seg_pos={0 0}, upload=233226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:33:28.467315314 (266h4m28.338398349s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:33:28.467322477 (266h4m28.338405522s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:33:28.799678512 (266h4m28.670761547s) [upload=233226]: Done uploading segments... Dec 16 17:33:28.799699542 (266h4m28.670782587s) [upload=233226]: Upload finished. err: <nil> Duration: 6.110788781s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [................XX....X...................XX................] duration: 3.511217723s trace: https://jaeger.dev.storj.io/trace/2d7e37a5218d47ae FAIL (2025-12-16 17:36:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.513129474s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................X......X......X....] duration: 5.8128954s trace: https://jaeger.dev.storj.io/trace/3a8b2eb64e78cdb7 FAIL (2025-12-16 17:35:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 17:33:25.825412431 (266h4m25.696495476s) [upload=233226]: Starting upload Dec 16 17:33:25.825716299 (266h4m25.696799334s) [upload=233226]: Done starting upload Dec 16 17:33:25.825900039 (266h4m25.696983074s) [upload=233226]: Uploading segments... Dec 16 17:33:25.825986874 (266h4m25.697069919s) [upload=233226]: Got next segment. Inline: false Dec 16 17:33:25.826024115 (266h4m25.69710716s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:33:25.826029405 (266h4m25.69711245s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:33:26.081528100 (266h4m25.952611145s) [seg_pos={0 0}, upload=233226]: Begin upload segment... Dec 16 17:33:26.081578937 (266h4m25.952661972s) [seg_pos={0 0}, upload=233226]: Done begin upload segment. Dec 16 17:33:26.081646916 (266h4m25.952729951s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.081689087 (266h4m25.952772122s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.082557751 (266h4m25.953640796s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083162883 (266h4m25.954245928s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083811628 (266h4m25.954894674s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.084268108 (266h4m25.955351153s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086136556 (266h4m25.957219591s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086764421 (266h4m25.957847456s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.087471016 (266h4m25.958554061s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.088841396 (266h4m25.959924442s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.089031668 (266h4m25.960114703s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.090373093 (266h4m25.961456128s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.092053242 (266h4m25.963136287s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.095858979 (266h4m25.966942024s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.096244182 (266h4m25.967327227s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.099898400 (266h4m25.970981445s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:27.793861758 (266h4m27.664944803s) [upload=233226]: Next returned nil segment Dec 16 17:33:27.793883430 (266h4m27.664966465s) [upload=233226]: Waiting for error group managing segments... Dec 16 17:33:28.076919891 (266h4m27.948002936s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.080532791 (266h4m27.951615826s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.081529298 (266h4m27.952612333s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.088220410 (266h4m27.959303455s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.102098585 (266h4m27.97318163s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.110557603 (266h4m27.981640648s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.112773352 (266h4m27.983856387s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.168421046 (266h4m28.039504091s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.175508041 (266h4m28.046591076s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.367477684 (266h4m28.23856073s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.388567654 (266h4m28.259650699s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.393640814 (266h4m28.264723859s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466522674 (266h4m28.337605719s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466547552 (266h4m28.337630597s) [seg_pos={0 0}, upload=233226]: Segment reached optimal threshold of 13 pieces. Dec 16 17:33:28.466920051 (266h4m28.338003096s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466932094 (266h4m28.338015129s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.466960198 (266h4m28.338043243s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466972481 (266h4m28.338055526s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467172281 (266h4m28.338255336s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection Dec 16 17:33:28.467182901 (266h4m28.338265946s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467216766 (266h4m28.338299811s) [seg_pos={0 0}, upload=233226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:33:28.467315314 (266h4m28.338398349s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:33:28.467322477 (266h4m28.338405522s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:33:28.799678512 (266h4m28.670761547s) [upload=233226]: Done uploading segments... Dec 16 17:33:28.799699542 (266h4m28.670782587s) [upload=233226]: Upload finished. err: <nil> Duration: 5.812919826s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................................X......X......X...] duration: 5.797766429s trace: https://jaeger.dev.storj.io/trace/3a8b2eb64e78cdb7 FAIL (2025-12-16 17:35:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 17:33:25.825412431 (266h4m25.696495476s) [upload=233226]: Starting upload Dec 16 17:33:25.825716299 (266h4m25.696799334s) [upload=233226]: Done starting upload Dec 16 17:33:25.825900039 (266h4m25.696983074s) [upload=233226]: Uploading segments... Dec 16 17:33:25.825986874 (266h4m25.697069919s) [upload=233226]: Got next segment. Inline: false Dec 16 17:33:25.826024115 (266h4m25.69710716s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:33:25.826029405 (266h4m25.69711245s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:33:26.081528100 (266h4m25.952611145s) [seg_pos={0 0}, upload=233226]: Begin upload segment... Dec 16 17:33:26.081578937 (266h4m25.952661972s) [seg_pos={0 0}, upload=233226]: Done begin upload segment. Dec 16 17:33:26.081646916 (266h4m25.952729951s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.081689087 (266h4m25.952772122s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.082557751 (266h4m25.953640796s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083162883 (266h4m25.954245928s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083811628 (266h4m25.954894674s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.084268108 (266h4m25.955351153s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086136556 (266h4m25.957219591s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086764421 (266h4m25.957847456s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.087471016 (266h4m25.958554061s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.088841396 (266h4m25.959924442s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.089031668 (266h4m25.960114703s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.090373093 (266h4m25.961456128s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.092053242 (266h4m25.963136287s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.095858979 (266h4m25.966942024s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.096244182 (266h4m25.967327227s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.099898400 (266h4m25.970981445s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:27.793861758 (266h4m27.664944803s) [upload=233226]: Next returned nil segment Dec 16 17:33:27.793883430 (266h4m27.664966465s) [upload=233226]: Waiting for error group managing segments... Dec 16 17:33:28.076919891 (266h4m27.948002936s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.080532791 (266h4m27.951615826s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.081529298 (266h4m27.952612333s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.088220410 (266h4m27.959303455s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.102098585 (266h4m27.97318163s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.110557603 (266h4m27.981640648s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.112773352 (266h4m27.983856387s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.168421046 (266h4m28.039504091s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.175508041 (266h4m28.046591076s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.367477684 (266h4m28.23856073s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.388567654 (266h4m28.259650699s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.393640814 (266h4m28.264723859s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466522674 (266h4m28.337605719s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466547552 (266h4m28.337630597s) [seg_pos={0 0}, upload=233226]: Segment reached optimal threshold of 13 pieces. Dec 16 17:33:28.466920051 (266h4m28.338003096s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466932094 (266h4m28.338015129s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.466960198 (266h4m28.338043243s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466972481 (266h4m28.338055526s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467172281 (266h4m28.338255336s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection Dec 16 17:33:28.467182901 (266h4m28.338265946s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467216766 (266h4m28.338299811s) [seg_pos={0 0}, upload=233226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:33:28.467315314 (266h4m28.338398349s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:33:28.467322477 (266h4m28.338405522s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:33:28.799678512 (266h4m28.670761547s) [upload=233226]: Done uploading segments... Dec 16 17:33:28.799699542 (266h4m28.670782587s) [upload=233226]: Upload finished. err: <nil> Duration: 5.797800875s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................................X......X......X..] duration: 6.061475598s trace: https://jaeger.dev.storj.io/trace/3a8b2eb64e78cdb7 FAIL (2025-12-16 17:35:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 17:33:25.825412431 (266h4m25.696495476s) [upload=233226]: Starting upload Dec 16 17:33:25.825716299 (266h4m25.696799334s) [upload=233226]: Done starting upload Dec 16 17:33:25.825900039 (266h4m25.696983074s) [upload=233226]: Uploading segments... Dec 16 17:33:25.825986874 (266h4m25.697069919s) [upload=233226]: Got next segment. Inline: false Dec 16 17:33:25.826024115 (266h4m25.69710716s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:33:25.826029405 (266h4m25.69711245s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:33:26.081528100 (266h4m25.952611145s) [seg_pos={0 0}, upload=233226]: Begin upload segment... Dec 16 17:33:26.081578937 (266h4m25.952661972s) [seg_pos={0 0}, upload=233226]: Done begin upload segment. Dec 16 17:33:26.081646916 (266h4m25.952729951s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.081689087 (266h4m25.952772122s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.082557751 (266h4m25.953640796s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083162883 (266h4m25.954245928s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083811628 (266h4m25.954894674s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.084268108 (266h4m25.955351153s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086136556 (266h4m25.957219591s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086764421 (266h4m25.957847456s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.087471016 (266h4m25.958554061s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.088841396 (266h4m25.959924442s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.089031668 (266h4m25.960114703s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.090373093 (266h4m25.961456128s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.092053242 (266h4m25.963136287s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.095858979 (266h4m25.966942024s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.096244182 (266h4m25.967327227s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.099898400 (266h4m25.970981445s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:27.793861758 (266h4m27.664944803s) [upload=233226]: Next returned nil segment Dec 16 17:33:27.793883430 (266h4m27.664966465s) [upload=233226]: Waiting for error group managing segments... Dec 16 17:33:28.076919891 (266h4m27.948002936s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.080532791 (266h4m27.951615826s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.081529298 (266h4m27.952612333s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.088220410 (266h4m27.959303455s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.102098585 (266h4m27.97318163s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.110557603 (266h4m27.981640648s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.112773352 (266h4m27.983856387s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.168421046 (266h4m28.039504091s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.175508041 (266h4m28.046591076s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.367477684 (266h4m28.23856073s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.388567654 (266h4m28.259650699s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.393640814 (266h4m28.264723859s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466522674 (266h4m28.337605719s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466547552 (266h4m28.337630597s) [seg_pos={0 0}, upload=233226]: Segment reached optimal threshold of 13 pieces. Dec 16 17:33:28.466920051 (266h4m28.338003096s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466932094 (266h4m28.338015129s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.466960198 (266h4m28.338043243s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466972481 (266h4m28.338055526s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467172281 (266h4m28.338255336s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection Dec 16 17:33:28.467182901 (266h4m28.338265946s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467216766 (266h4m28.338299811s) [seg_pos={0 0}, upload=233226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:33:28.467315314 (266h4m28.338398349s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:33:28.467322477 (266h4m28.338405522s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:33:28.799678512 (266h4m28.670761547s) [upload=233226]: Done uploading segments... Dec 16 17:33:28.799699542 (266h4m28.670782587s) [upload=233226]: Upload finished. err: <nil> Duration: 6.061507217s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [X................XX....X...................XX...............] duration: 3.586963737s trace: https://jaeger.dev.storj.io/trace/2d7e37a5218d47ae FAIL (2025-12-16 17:35:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.586984316s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................................X......X......X.] duration: 6.074979541s trace: https://jaeger.dev.storj.io/trace/3a8b2eb64e78cdb7 FAIL (2025-12-16 17:34:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 17:33:25.825412431 (266h4m25.696495476s) [upload=233226]: Starting upload Dec 16 17:33:25.825716299 (266h4m25.696799334s) [upload=233226]: Done starting upload Dec 16 17:33:25.825900039 (266h4m25.696983074s) [upload=233226]: Uploading segments... Dec 16 17:33:25.825986874 (266h4m25.697069919s) [upload=233226]: Got next segment. Inline: false Dec 16 17:33:25.826024115 (266h4m25.69710716s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:33:25.826029405 (266h4m25.69711245s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:33:26.081528100 (266h4m25.952611145s) [seg_pos={0 0}, upload=233226]: Begin upload segment... Dec 16 17:33:26.081578937 (266h4m25.952661972s) [seg_pos={0 0}, upload=233226]: Done begin upload segment. Dec 16 17:33:26.081646916 (266h4m25.952729951s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.081689087 (266h4m25.952772122s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.082557751 (266h4m25.953640796s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083162883 (266h4m25.954245928s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083811628 (266h4m25.954894674s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.084268108 (266h4m25.955351153s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086136556 (266h4m25.957219591s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086764421 (266h4m25.957847456s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.087471016 (266h4m25.958554061s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.088841396 (266h4m25.959924442s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.089031668 (266h4m25.960114703s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.090373093 (266h4m25.961456128s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.092053242 (266h4m25.963136287s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.095858979 (266h4m25.966942024s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.096244182 (266h4m25.967327227s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.099898400 (266h4m25.970981445s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:27.793861758 (266h4m27.664944803s) [upload=233226]: Next returned nil segment Dec 16 17:33:27.793883430 (266h4m27.664966465s) [upload=233226]: Waiting for error group managing segments... Dec 16 17:33:28.076919891 (266h4m27.948002936s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.080532791 (266h4m27.951615826s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.081529298 (266h4m27.952612333s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.088220410 (266h4m27.959303455s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.102098585 (266h4m27.97318163s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.110557603 (266h4m27.981640648s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.112773352 (266h4m27.983856387s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.168421046 (266h4m28.039504091s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.175508041 (266h4m28.046591076s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.367477684 (266h4m28.23856073s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.388567654 (266h4m28.259650699s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.393640814 (266h4m28.264723859s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466522674 (266h4m28.337605719s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466547552 (266h4m28.337630597s) [seg_pos={0 0}, upload=233226]: Segment reached optimal threshold of 13 pieces. Dec 16 17:33:28.466920051 (266h4m28.338003096s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466932094 (266h4m28.338015129s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.466960198 (266h4m28.338043243s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466972481 (266h4m28.338055526s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467172281 (266h4m28.338255336s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection Dec 16 17:33:28.467182901 (266h4m28.338265946s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467216766 (266h4m28.338299811s) [seg_pos={0 0}, upload=233226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:33:28.467315314 (266h4m28.338398349s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:33:28.467322477 (266h4m28.338405522s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:33:28.799678512 (266h4m28.670761547s) [upload=233226]: Done uploading segments... Dec 16 17:33:28.799699542 (266h4m28.670782587s) [upload=233226]: Upload finished. err: <nil> Duration: 6.075010339s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................................X......X......X] duration: 30.002523513s trace: https://jaeger.dev.storj.io/trace/3a8b2eb64e78cdb7 FAIL (2025-12-16 17:33:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 17:33:25.825412431 (266h4m25.696495476s) [upload=233226]: Starting upload Dec 16 17:33:25.825716299 (266h4m25.696799334s) [upload=233226]: Done starting upload Dec 16 17:33:25.825900039 (266h4m25.696983074s) [upload=233226]: Uploading segments... Dec 16 17:33:25.825986874 (266h4m25.697069919s) [upload=233226]: Got next segment. Inline: false Dec 16 17:33:25.826024115 (266h4m25.69710716s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:33:25.826029405 (266h4m25.69711245s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:33:26.081528100 (266h4m25.952611145s) [seg_pos={0 0}, upload=233226]: Begin upload segment... Dec 16 17:33:26.081578937 (266h4m25.952661972s) [seg_pos={0 0}, upload=233226]: Done begin upload segment. Dec 16 17:33:26.081646916 (266h4m25.952729951s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.081689087 (266h4m25.952772122s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.082557751 (266h4m25.953640796s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083162883 (266h4m25.954245928s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083811628 (266h4m25.954894674s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.084268108 (266h4m25.955351153s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086136556 (266h4m25.957219591s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086764421 (266h4m25.957847456s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.087471016 (266h4m25.958554061s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.088841396 (266h4m25.959924442s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.089031668 (266h4m25.960114703s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.090373093 (266h4m25.961456128s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.092053242 (266h4m25.963136287s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.095858979 (266h4m25.966942024s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.096244182 (266h4m25.967327227s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.099898400 (266h4m25.970981445s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:27.793861758 (266h4m27.664944803s) [upload=233226]: Next returned nil segment Dec 16 17:33:27.793883430 (266h4m27.664966465s) [upload=233226]: Waiting for error group managing segments... Dec 16 17:33:28.076919891 (266h4m27.948002936s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.080532791 (266h4m27.951615826s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.081529298 (266h4m27.952612333s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.088220410 (266h4m27.959303455s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.102098585 (266h4m27.97318163s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.110557603 (266h4m27.981640648s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.112773352 (266h4m27.983856387s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.168421046 (266h4m28.039504091s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.175508041 (266h4m28.046591076s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.367477684 (266h4m28.23856073s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.388567654 (266h4m28.259650699s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.393640814 (266h4m28.264723859s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466522674 (266h4m28.337605719s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466547552 (266h4m28.337630597s) [seg_pos={0 0}, upload=233226]: Segment reached optimal threshold of 13 pieces. Dec 16 17:33:28.466920051 (266h4m28.338003096s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466932094 (266h4m28.338015129s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.466960198 (266h4m28.338043243s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466972481 (266h4m28.338055526s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467172281 (266h4m28.338255336s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection Dec 16 17:33:28.467182901 (266h4m28.338265946s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467216766 (266h4m28.338299811s) [seg_pos={0 0}, upload=233226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:33:28.467315314 (266h4m28.338398349s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:33:28.467322477 (266h4m28.338405522s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:33:28.799678512 (266h4m28.670761547s) [upload=233226]: Done uploading segments... Dec 16 17:33:28.799699542 (266h4m28.670782587s) [upload=233226]: Upload finished. err: <nil> Duration: 30.002546756s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.X................XX....X...................XX..............] duration: 4.077270208s trace: https://jaeger.dev.storj.io/trace/2d7e37a5218d47ae FAIL (2025-12-16 17:33:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.077298431s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X......X......] duration: 5.724994474s trace: https://jaeger.dev.storj.io/trace/1693cf00b42bb85 FAIL (2025-12-16 17:33:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: MVLHGEWQOWMJ6PZ4Y6EA7ALQFHGWGJFAGTPBJZLEKIY3PZP4W5YQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: VROU556GWHXECJ24OIAZQN7EB37RG4YY65EE3MPHDUZWOH3QBSEA) context deadline exceeded ======LOGS====== Dec 16 17:28:42.970569461 (265h59m42.841652506s) [upload=233159]: Starting upload Dec 16 17:28:42.970742811 (265h59m42.841825856s) [upload=233159]: Done starting upload Dec 16 17:28:42.970820098 (265h59m42.841903133s) [upload=233159]: Uploading segments... Dec 16 17:28:42.971016382 (265h59m42.842099417s) [upload=233159]: Got next segment. Inline: false Dec 16 17:28:42.971071286 (265h59m42.842154321s) [upload=233159]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:28:42.971076878 (265h59m42.842159923s) [upload=233159]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:28:43.215333084 (265h59m43.086416119s) [seg_pos={0 0}, upload=233159]: Begin upload segment... Dec 16 17:28:43.215388780 (265h59m43.086471816s) [seg_pos={0 0}, upload=233159]: Done begin upload segment. Dec 16 17:28:43.215416844 (265h59m43.086499879s) [noise=true , address=145.79.227.6:4258 , piece_id=TTPW2CX4XVWMJZV4J6JW4MRSRIGLW7Y6PEXCEYIDUNIOLM3AOYPA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.215505913 (265h59m43.086588958s) [noise=true , address=45.91.116.5:4300 , piece_id=2C67D3NKKHRBO3J26M3PTZTQLN7DURXY2KZG5A2YKAXCMQRIWJNQ, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.216114753 (265h59m43.087197798s) [noise=true , address=145.79.227.40:4049 , piece_id=SXWS3NSWYZVVXF2GWOOBKTPYRZQI4PICQJA2XD7NPAHCQGKZT6IA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.216423660 (265h59m43.087506695s) [noise=true , address=145.79.227.42:4121 , piece_id=LXGGY5STJ5UOEPYRNESNFZZ55UZY5DNYNV4PCGXSIK22KDCWUHGA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.216904276 (265h59m43.087987321s) [noise=true , address=77.90.31.11:4556 , piece_id=7V22BCWLKH422EKA3DJYUJPCAQ5LSTQRY5KB67VABMYWCXZI2MQA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.217406693 (265h59m43.088489738s) [noise=true , address=145.79.227.39:4015 , piece_id=JE2O7EODSUOO6GG4QDGR7RGQNOWY4ZCQNQ2T6RK7K26X667RMI2A, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.217976387 (265h59m43.089059422s) [noise=true , address=77.90.31.14:4664 , piece_id=VROU556GWHXECJ24OIAZQN7EB37RG4YY65EE3MPHDUZWOH3QBSEA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.219456156 (265h59m43.090539191s) [noise=true , address=145.79.227.8:4330 , piece_id=M44U54AVO6GBJA7P2ZKQGT4MJFQG7EM4CMNHFP6AMLNOV3APKVEA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.219657959 (265h59m43.090740994s) [noise=true , address=77.90.31.13:4629 , piece_id=MVLHGEWQOWMJ6PZ4Y6EA7ALQFHGWGJFAGTPBJZLEKIY3PZP4W5YQ, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.220950371 (265h59m43.092033406s) [noise=true , address=77.90.31.10:4520 , piece_id=HB5OCNUMFQ6X3UMEOQDSYPC2A3VNQTR7LJYZ3AOMJS3PVSIVLHGA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.221108251 (265h59m43.092191286s) [noise=true , address=145.79.227.7:4293 , piece_id=NANZ2FGRTALFYHJSFU5UWULLNIRMK4OGHAMO3GXH4PDRCAHUPPRA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.222409148 (265h59m43.093492183s) [noise=true , address=145.79.227.26:4655 , piece_id=D4BD3GGRPY7KRQXEJGEPEROBL6UTWE4ME36CHZM3UWF7C6KXSV5Q, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.224018112 (265h59m43.095101147s) [noise=true , address=145.79.227.50:4375 , piece_id=63ZC5AKNL6LNH7OIYODD7P5QXC7H6MVQX2IIEKUQV4BAFJMULAFA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.228049598 (265h59m43.099132673s) [noise=true , address=145.79.227.27:4689 , piece_id=ZSB47AEORTP64FGGNZG6LVCE2JPJYYXCZLL6LVVOGSNBSNTCFDHA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.230997011 (265h59m43.102080046s) [noise=true , address=145.79.227.32:4834 , piece_id=XTSKBNNSNCRT2I3ZFGNS7FNWUPDTHTVRJRRX4MSM4PYO7Y7Q3F4Q, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.233277594 (265h59m43.104360639s) [noise=true , address=77.90.31.19:4842 , piece_id=7UDB5VJP35YSCK6BZ7SB2IQC4QR7GDYXWTBDS2IAULL4C63VNHTQ, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:44.922178743 (265h59m44.793261788s) [upload=233159]: Next returned nil segment Dec 16 17:28:44.922194903 (265h59m44.793277948s) [upload=233159]: Waiting for error group managing segments... Dec 16 17:28:45.208110307 (265h59m45.079193352s) [noise=true , address=145.79.227.39:4015 , piece_id=JE2O7EODSUOO6GG4QDGR7RGQNOWY4ZCQNQ2T6RK7K26X667RMI2A, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.223090081 (265h59m45.094173116s) [noise=true , address=145.79.227.6:4258 , piece_id=TTPW2CX4XVWMJZV4J6JW4MRSRIGLW7Y6PEXCEYIDUNIOLM3AOYPA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.223193477 (265h59m45.094276513s) [noise=true , address=145.79.227.50:4375 , piece_id=63ZC5AKNL6LNH7OIYODD7P5QXC7H6MVQX2IIEKUQV4BAFJMULAFA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.235649686 (265h59m45.106732731s) [noise=true , address=145.79.227.8:4330 , piece_id=M44U54AVO6GBJA7P2ZKQGT4MJFQG7EM4CMNHFP6AMLNOV3APKVEA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.250515143 (265h59m45.121598178s) [noise=true , address=145.79.227.32:4834 , piece_id=XTSKBNNSNCRT2I3ZFGNS7FNWUPDTHTVRJRRX4MSM4PYO7Y7Q3F4Q, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.257911507 (265h59m45.128994542s) [noise=true , address=145.79.227.42:4121 , piece_id=LXGGY5STJ5UOEPYRNESNFZZ55UZY5DNYNV4PCGXSIK22KDCWUHGA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.258894539 (265h59m45.129977574s) [noise=true , address=145.79.227.40:4049 , piece_id=SXWS3NSWYZVVXF2GWOOBKTPYRZQI4PICQJA2XD7NPAHCQGKZT6IA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.286276720 (265h59m45.157359765s) [noise=true , address=145.79.227.7:4293 , piece_id=NANZ2FGRTALFYHJSFU5UWULLNIRMK4OGHAMO3GXH4PDRCAHUPPRA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.289714986 (265h59m45.160798031s) [noise=true , address=145.79.227.26:4655 , piece_id=D4BD3GGRPY7KRQXEJGEPEROBL6UTWE4ME36CHZM3UWF7C6KXSV5Q, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.300709662 (265h59m45.171792707s) [noise=true , address=145.79.227.27:4689 , piece_id=ZSB47AEORTP64FGGNZG6LVCE2JPJYYXCZLL6LVVOGSNBSNTCFDHA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.404659822 (265h59m45.275742857s) [noise=true , address=77.90.31.13:4629 , piece_id=MVLHGEWQOWMJ6PZ4Y6EA7ALQFHGWGJFAGTPBJZLEKIY3PZP4W5YQ, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.478316668 (265h59m45.349399713s) [noise=true , address=77.90.31.19:4842 , piece_id=7UDB5VJP35YSCK6BZ7SB2IQC4QR7GDYXWTBDS2IAULL4C63VNHTQ, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.516995660 (265h59m45.388078705s) [noise=true , address=77.90.31.14:4664 , piece_id=VROU556GWHXECJ24OIAZQN7EB37RG4YY65EE3MPHDUZWOH3QBSEA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.517022651 (265h59m45.388105686s) [seg_pos={0 0}, upload=233159]: Segment reached optimal threshold of 13 pieces. Dec 16 17:28:45.517288747 (265h59m45.388371812s) [noise=true , address=77.90.31.10:4520 , piece_id=HB5OCNUMFQ6X3UMEOQDSYPC2A3VNQTR7LJYZ3AOMJS3PVSIVLHGA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 16 17:28:45.517298115 (265h59m45.38838115s) [noise=true , address=77.90.31.10:4520 , piece_id=HB5OCNUMFQ6X3UMEOQDSYPC2A3VNQTR7LJYZ3AOMJS3PVSIVLHGA, seg_pos={0 0}, upload=233159]: Long tail cancellation due to optimal threshold Dec 16 17:28:45.517440477 (265h59m45.388523542s) [noise=true , address=45.91.116.5:4300 , piece_id=2C67D3NKKHRBO3J26M3PTZTQLN7DURXY2KZG5A2YKAXCMQRIWJNQ, seg_pos={0 0}, upload=233159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 16 17:28:45.517463720 (265h59m45.388546775s) [noise=true , address=45.91.116.5:4300 , piece_id=2C67D3NKKHRBO3J26M3PTZTQLN7DURXY2KZG5A2YKAXCMQRIWJNQ, seg_pos={0 0}, upload=233159]: Long tail cancellation due to optimal threshold Dec 16 17:28:45.517602394 (265h59m45.38868544s) [noise=true , address=77.90.31.11:4556 , piece_id=7V22BCWLKH422EKA3DJYUJPCAQ5LSTQRY5KB67VABMYWCXZI2MQA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 16 17:28:45.517612263 (265h59m45.388695298s) [noise=true , address=77.90.31.11:4556 , piece_id=7V22BCWLKH422EKA3DJYUJPCAQ5LSTQRY5KB67VABMYWCXZI2MQA, seg_pos={0 0}, upload=233159]: Long tail cancellation due to optimal threshold Dec 16 17:28:45.517640707 (265h59m45.388723752s) [seg_pos={0 0}, upload=233159]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:28:45.517730238 (265h59m45.388813283s) [upload=233159]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:28:45.517736550 (265h59m45.388819595s) [upload=233159]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:28:45.806056282 (265h59m45.677139317s) [upload=233159]: Done uploading segments... Dec 16 17:28:45.806074638 (265h59m45.677157673s) [upload=233159]: Upload finished. err: <nil> Duration: 5.725021497s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..X................XX....X...................XX.............] duration: 3.488938961s trace: https://jaeger.dev.storj.io/trace/2d7e37a5218d47ae FAIL (2025-12-16 17:32:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.488966162s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...X................XX....X...................XX............] duration: 3.596395147s trace: https://jaeger.dev.storj.io/trace/2d7e37a5218d47ae FAIL (2025-12-16 17:32:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.59642293s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X......X.....] duration: 5.746156736s trace: https://jaeger.dev.storj.io/trace/1693cf00b42bb85 FAIL (2025-12-16 17:32:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: MVLHGEWQOWMJ6PZ4Y6EA7ALQFHGWGJFAGTPBJZLEKIY3PZP4W5YQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: VROU556GWHXECJ24OIAZQN7EB37RG4YY65EE3MPHDUZWOH3QBSEA) context deadline exceeded ======LOGS====== Dec 16 17:28:42.970569461 (265h59m42.841652506s) [upload=233159]: Starting upload Dec 16 17:28:42.970742811 (265h59m42.841825856s) [upload=233159]: Done starting upload Dec 16 17:28:42.970820098 (265h59m42.841903133s) [upload=233159]: Uploading segments... Dec 16 17:28:42.971016382 (265h59m42.842099417s) [upload=233159]: Got next segment. Inline: false Dec 16 17:28:42.971071286 (265h59m42.842154321s) [upload=233159]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:28:42.971076878 (265h59m42.842159923s) [upload=233159]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:28:43.215333084 (265h59m43.086416119s) [seg_pos={0 0}, upload=233159]: Begin upload segment... Dec 16 17:28:43.215388780 (265h59m43.086471816s) [seg_pos={0 0}, upload=233159]: Done begin upload segment. Dec 16 17:28:43.215416844 (265h59m43.086499879s) [noise=true , address=145.79.227.6:4258 , piece_id=TTPW2CX4XVWMJZV4J6JW4MRSRIGLW7Y6PEXCEYIDUNIOLM3AOYPA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.215505913 (265h59m43.086588958s) [noise=true , address=45.91.116.5:4300 , piece_id=2C67D3NKKHRBO3J26M3PTZTQLN7DURXY2KZG5A2YKAXCMQRIWJNQ, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.216114753 (265h59m43.087197798s) [noise=true , address=145.79.227.40:4049 , piece_id=SXWS3NSWYZVVXF2GWOOBKTPYRZQI4PICQJA2XD7NPAHCQGKZT6IA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.216423660 (265h59m43.087506695s) [noise=true , address=145.79.227.42:4121 , piece_id=LXGGY5STJ5UOEPYRNESNFZZ55UZY5DNYNV4PCGXSIK22KDCWUHGA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.216904276 (265h59m43.087987321s) [noise=true , address=77.90.31.11:4556 , piece_id=7V22BCWLKH422EKA3DJYUJPCAQ5LSTQRY5KB67VABMYWCXZI2MQA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.217406693 (265h59m43.088489738s) [noise=true , address=145.79.227.39:4015 , piece_id=JE2O7EODSUOO6GG4QDGR7RGQNOWY4ZCQNQ2T6RK7K26X667RMI2A, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.217976387 (265h59m43.089059422s) [noise=true , address=77.90.31.14:4664 , piece_id=VROU556GWHXECJ24OIAZQN7EB37RG4YY65EE3MPHDUZWOH3QBSEA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.219456156 (265h59m43.090539191s) [noise=true , address=145.79.227.8:4330 , piece_id=M44U54AVO6GBJA7P2ZKQGT4MJFQG7EM4CMNHFP6AMLNOV3APKVEA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.219657959 (265h59m43.090740994s) [noise=true , address=77.90.31.13:4629 , piece_id=MVLHGEWQOWMJ6PZ4Y6EA7ALQFHGWGJFAGTPBJZLEKIY3PZP4W5YQ, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.220950371 (265h59m43.092033406s) [noise=true , address=77.90.31.10:4520 , piece_id=HB5OCNUMFQ6X3UMEOQDSYPC2A3VNQTR7LJYZ3AOMJS3PVSIVLHGA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.221108251 (265h59m43.092191286s) [noise=true , address=145.79.227.7:4293 , piece_id=NANZ2FGRTALFYHJSFU5UWULLNIRMK4OGHAMO3GXH4PDRCAHUPPRA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.222409148 (265h59m43.093492183s) [noise=true , address=145.79.227.26:4655 , piece_id=D4BD3GGRPY7KRQXEJGEPEROBL6UTWE4ME36CHZM3UWF7C6KXSV5Q, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.224018112 (265h59m43.095101147s) [noise=true , address=145.79.227.50:4375 , piece_id=63ZC5AKNL6LNH7OIYODD7P5QXC7H6MVQX2IIEKUQV4BAFJMULAFA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.228049598 (265h59m43.099132673s) [noise=true , address=145.79.227.27:4689 , piece_id=ZSB47AEORTP64FGGNZG6LVCE2JPJYYXCZLL6LVVOGSNBSNTCFDHA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.230997011 (265h59m43.102080046s) [noise=true , address=145.79.227.32:4834 , piece_id=XTSKBNNSNCRT2I3ZFGNS7FNWUPDTHTVRJRRX4MSM4PYO7Y7Q3F4Q, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.233277594 (265h59m43.104360639s) [noise=true , address=77.90.31.19:4842 , piece_id=7UDB5VJP35YSCK6BZ7SB2IQC4QR7GDYXWTBDS2IAULL4C63VNHTQ, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:44.922178743 (265h59m44.793261788s) [upload=233159]: Next returned nil segment Dec 16 17:28:44.922194903 (265h59m44.793277948s) [upload=233159]: Waiting for error group managing segments... Dec 16 17:28:45.208110307 (265h59m45.079193352s) [noise=true , address=145.79.227.39:4015 , piece_id=JE2O7EODSUOO6GG4QDGR7RGQNOWY4ZCQNQ2T6RK7K26X667RMI2A, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.223090081 (265h59m45.094173116s) [noise=true , address=145.79.227.6:4258 , piece_id=TTPW2CX4XVWMJZV4J6JW4MRSRIGLW7Y6PEXCEYIDUNIOLM3AOYPA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.223193477 (265h59m45.094276513s) [noise=true , address=145.79.227.50:4375 , piece_id=63ZC5AKNL6LNH7OIYODD7P5QXC7H6MVQX2IIEKUQV4BAFJMULAFA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.235649686 (265h59m45.106732731s) [noise=true , address=145.79.227.8:4330 , piece_id=M44U54AVO6GBJA7P2ZKQGT4MJFQG7EM4CMNHFP6AMLNOV3APKVEA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.250515143 (265h59m45.121598178s) [noise=true , address=145.79.227.32:4834 , piece_id=XTSKBNNSNCRT2I3ZFGNS7FNWUPDTHTVRJRRX4MSM4PYO7Y7Q3F4Q, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.257911507 (265h59m45.128994542s) [noise=true , address=145.79.227.42:4121 , piece_id=LXGGY5STJ5UOEPYRNESNFZZ55UZY5DNYNV4PCGXSIK22KDCWUHGA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.258894539 (265h59m45.129977574s) [noise=true , address=145.79.227.40:4049 , piece_id=SXWS3NSWYZVVXF2GWOOBKTPYRZQI4PICQJA2XD7NPAHCQGKZT6IA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.286276720 (265h59m45.157359765s) [noise=true , address=145.79.227.7:4293 , piece_id=NANZ2FGRTALFYHJSFU5UWULLNIRMK4OGHAMO3GXH4PDRCAHUPPRA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.289714986 (265h59m45.160798031s) [noise=true , address=145.79.227.26:4655 , piece_id=D4BD3GGRPY7KRQXEJGEPEROBL6UTWE4ME36CHZM3UWF7C6KXSV5Q, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.300709662 (265h59m45.171792707s) [noise=true , address=145.79.227.27:4689 , piece_id=ZSB47AEORTP64FGGNZG6LVCE2JPJYYXCZLL6LVVOGSNBSNTCFDHA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.404659822 (265h59m45.275742857s) [noise=true , address=77.90.31.13:4629 , piece_id=MVLHGEWQOWMJ6PZ4Y6EA7ALQFHGWGJFAGTPBJZLEKIY3PZP4W5YQ, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.478316668 (265h59m45.349399713s) [noise=true , address=77.90.31.19:4842 , piece_id=7UDB5VJP35YSCK6BZ7SB2IQC4QR7GDYXWTBDS2IAULL4C63VNHTQ, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.516995660 (265h59m45.388078705s) [noise=true , address=77.90.31.14:4664 , piece_id=VROU556GWHXECJ24OIAZQN7EB37RG4YY65EE3MPHDUZWOH3QBSEA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.517022651 (265h59m45.388105686s) [seg_pos={0 0}, upload=233159]: Segment reached optimal threshold of 13 pieces. Dec 16 17:28:45.517288747 (265h59m45.388371812s) [noise=true , address=77.90.31.10:4520 , piece_id=HB5OCNUMFQ6X3UMEOQDSYPC2A3VNQTR7LJYZ3AOMJS3PVSIVLHGA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 16 17:28:45.517298115 (265h59m45.38838115s) [noise=true , address=77.90.31.10:4520 , piece_id=HB5OCNUMFQ6X3UMEOQDSYPC2A3VNQTR7LJYZ3AOMJS3PVSIVLHGA, seg_pos={0 0}, upload=233159]: Long tail cancellation due to optimal threshold Dec 16 17:28:45.517440477 (265h59m45.388523542s) [noise=true , address=45.91.116.5:4300 , piece_id=2C67D3NKKHRBO3J26M3PTZTQLN7DURXY2KZG5A2YKAXCMQRIWJNQ, seg_pos={0 0}, upload=233159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 16 17:28:45.517463720 (265h59m45.388546775s) [noise=true , address=45.91.116.5:4300 , piece_id=2C67D3NKKHRBO3J26M3PTZTQLN7DURXY2KZG5A2YKAXCMQRIWJNQ, seg_pos={0 0}, upload=233159]: Long tail cancellation due to optimal threshold Dec 16 17:28:45.517602394 (265h59m45.38868544s) [noise=true , address=77.90.31.11:4556 , piece_id=7V22BCWLKH422EKA3DJYUJPCAQ5LSTQRY5KB67VABMYWCXZI2MQA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 16 17:28:45.517612263 (265h59m45.388695298s) [noise=true , address=77.90.31.11:4556 , piece_id=7V22BCWLKH422EKA3DJYUJPCAQ5LSTQRY5KB67VABMYWCXZI2MQA, seg_pos={0 0}, upload=233159]: Long tail cancellation due to optimal threshold Dec 16 17:28:45.517640707 (265h59m45.388723752s) [seg_pos={0 0}, upload=233159]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:28:45.517730238 (265h59m45.388813283s) [upload=233159]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:28:45.517736550 (265h59m45.388819595s) [upload=233159]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:28:45.806056282 (265h59m45.677139317s) [upload=233159]: Done uploading segments... Dec 16 17:28:45.806074638 (265h59m45.677157673s) [upload=233159]: Upload finished. err: <nil> Duration: 5.746189749s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [....X................XX....X...................XX...........] duration: 3.409815485s trace: https://jaeger.dev.storj.io/trace/2d7e37a5218d47ae FAIL (2025-12-16 17:31:55 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.409850321s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X......X....] duration: 5.856566264s trace: https://jaeger.dev.storj.io/trace/1693cf00b42bb85 FAIL (2025-12-16 17:31:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: MVLHGEWQOWMJ6PZ4Y6EA7ALQFHGWGJFAGTPBJZLEKIY3PZP4W5YQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: VROU556GWHXECJ24OIAZQN7EB37RG4YY65EE3MPHDUZWOH3QBSEA) context deadline exceeded ======LOGS====== Dec 16 17:28:42.970569461 (265h59m42.841652506s) [upload=233159]: Starting upload Dec 16 17:28:42.970742811 (265h59m42.841825856s) [upload=233159]: Done starting upload Dec 16 17:28:42.970820098 (265h59m42.841903133s) [upload=233159]: Uploading segments... Dec 16 17:28:42.971016382 (265h59m42.842099417s) [upload=233159]: Got next segment. Inline: false Dec 16 17:28:42.971071286 (265h59m42.842154321s) [upload=233159]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:28:42.971076878 (265h59m42.842159923s) [upload=233159]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:28:43.215333084 (265h59m43.086416119s) [seg_pos={0 0}, upload=233159]: Begin upload segment... Dec 16 17:28:43.215388780 (265h59m43.086471816s) [seg_pos={0 0}, upload=233159]: Done begin upload segment. Dec 16 17:28:43.215416844 (265h59m43.086499879s) [noise=true , address=145.79.227.6:4258 , piece_id=TTPW2CX4XVWMJZV4J6JW4MRSRIGLW7Y6PEXCEYIDUNIOLM3AOYPA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.215505913 (265h59m43.086588958s) [noise=true , address=45.91.116.5:4300 , piece_id=2C67D3NKKHRBO3J26M3PTZTQLN7DURXY2KZG5A2YKAXCMQRIWJNQ, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.216114753 (265h59m43.087197798s) [noise=true , address=145.79.227.40:4049 , piece_id=SXWS3NSWYZVVXF2GWOOBKTPYRZQI4PICQJA2XD7NPAHCQGKZT6IA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.216423660 (265h59m43.087506695s) [noise=true , address=145.79.227.42:4121 , piece_id=LXGGY5STJ5UOEPYRNESNFZZ55UZY5DNYNV4PCGXSIK22KDCWUHGA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.216904276 (265h59m43.087987321s) [noise=true , address=77.90.31.11:4556 , piece_id=7V22BCWLKH422EKA3DJYUJPCAQ5LSTQRY5KB67VABMYWCXZI2MQA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.217406693 (265h59m43.088489738s) [noise=true , address=145.79.227.39:4015 , piece_id=JE2O7EODSUOO6GG4QDGR7RGQNOWY4ZCQNQ2T6RK7K26X667RMI2A, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.217976387 (265h59m43.089059422s) [noise=true , address=77.90.31.14:4664 , piece_id=VROU556GWHXECJ24OIAZQN7EB37RG4YY65EE3MPHDUZWOH3QBSEA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.219456156 (265h59m43.090539191s) [noise=true , address=145.79.227.8:4330 , piece_id=M44U54AVO6GBJA7P2ZKQGT4MJFQG7EM4CMNHFP6AMLNOV3APKVEA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.219657959 (265h59m43.090740994s) [noise=true , address=77.90.31.13:4629 , piece_id=MVLHGEWQOWMJ6PZ4Y6EA7ALQFHGWGJFAGTPBJZLEKIY3PZP4W5YQ, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.220950371 (265h59m43.092033406s) [noise=true , address=77.90.31.10:4520 , piece_id=HB5OCNUMFQ6X3UMEOQDSYPC2A3VNQTR7LJYZ3AOMJS3PVSIVLHGA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.221108251 (265h59m43.092191286s) [noise=true , address=145.79.227.7:4293 , piece_id=NANZ2FGRTALFYHJSFU5UWULLNIRMK4OGHAMO3GXH4PDRCAHUPPRA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.222409148 (265h59m43.093492183s) [noise=true , address=145.79.227.26:4655 , piece_id=D4BD3GGRPY7KRQXEJGEPEROBL6UTWE4ME36CHZM3UWF7C6KXSV5Q, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.224018112 (265h59m43.095101147s) [noise=true , address=145.79.227.50:4375 , piece_id=63ZC5AKNL6LNH7OIYODD7P5QXC7H6MVQX2IIEKUQV4BAFJMULAFA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.228049598 (265h59m43.099132673s) [noise=true , address=145.79.227.27:4689 , piece_id=ZSB47AEORTP64FGGNZG6LVCE2JPJYYXCZLL6LVVOGSNBSNTCFDHA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.230997011 (265h59m43.102080046s) [noise=true , address=145.79.227.32:4834 , piece_id=XTSKBNNSNCRT2I3ZFGNS7FNWUPDTHTVRJRRX4MSM4PYO7Y7Q3F4Q, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.233277594 (265h59m43.104360639s) [noise=true , address=77.90.31.19:4842 , piece_id=7UDB5VJP35YSCK6BZ7SB2IQC4QR7GDYXWTBDS2IAULL4C63VNHTQ, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:44.922178743 (265h59m44.793261788s) [upload=233159]: Next returned nil segment Dec 16 17:28:44.922194903 (265h59m44.793277948s) [upload=233159]: Waiting for error group managing segments... Dec 16 17:28:45.208110307 (265h59m45.079193352s) [noise=true , address=145.79.227.39:4015 , piece_id=JE2O7EODSUOO6GG4QDGR7RGQNOWY4ZCQNQ2T6RK7K26X667RMI2A, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.223090081 (265h59m45.094173116s) [noise=true , address=145.79.227.6:4258 , piece_id=TTPW2CX4XVWMJZV4J6JW4MRSRIGLW7Y6PEXCEYIDUNIOLM3AOYPA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.223193477 (265h59m45.094276513s) [noise=true , address=145.79.227.50:4375 , piece_id=63ZC5AKNL6LNH7OIYODD7P5QXC7H6MVQX2IIEKUQV4BAFJMULAFA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.235649686 (265h59m45.106732731s) [noise=true , address=145.79.227.8:4330 , piece_id=M44U54AVO6GBJA7P2ZKQGT4MJFQG7EM4CMNHFP6AMLNOV3APKVEA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.250515143 (265h59m45.121598178s) [noise=true , address=145.79.227.32:4834 , piece_id=XTSKBNNSNCRT2I3ZFGNS7FNWUPDTHTVRJRRX4MSM4PYO7Y7Q3F4Q, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.257911507 (265h59m45.128994542s) [noise=true , address=145.79.227.42:4121 , piece_id=LXGGY5STJ5UOEPYRNESNFZZ55UZY5DNYNV4PCGXSIK22KDCWUHGA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.258894539 (265h59m45.129977574s) [noise=true , address=145.79.227.40:4049 , piece_id=SXWS3NSWYZVVXF2GWOOBKTPYRZQI4PICQJA2XD7NPAHCQGKZT6IA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.286276720 (265h59m45.157359765s) [noise=true , address=145.79.227.7:4293 , piece_id=NANZ2FGRTALFYHJSFU5UWULLNIRMK4OGHAMO3GXH4PDRCAHUPPRA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.289714986 (265h59m45.160798031s) [noise=true , address=145.79.227.26:4655 , piece_id=D4BD3GGRPY7KRQXEJGEPEROBL6UTWE4ME36CHZM3UWF7C6KXSV5Q, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.300709662 (265h59m45.171792707s) [noise=true , address=145.79.227.27:4689 , piece_id=ZSB47AEORTP64FGGNZG6LVCE2JPJYYXCZLL6LVVOGSNBSNTCFDHA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.404659822 (265h59m45.275742857s) [noise=true , address=77.90.31.13:4629 , piece_id=MVLHGEWQOWMJ6PZ4Y6EA7ALQFHGWGJFAGTPBJZLEKIY3PZP4W5YQ, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.478316668 (265h59m45.349399713s) [noise=true , address=77.90.31.19:4842 , piece_id=7UDB5VJP35YSCK6BZ7SB2IQC4QR7GDYXWTBDS2IAULL4C63VNHTQ, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.516995660 (265h59m45.388078705s) [noise=true , address=77.90.31.14:4664 , piece_id=VROU556GWHXECJ24OIAZQN7EB37RG4YY65EE3MPHDUZWOH3QBSEA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.517022651 (265h59m45.388105686s) [seg_pos={0 0}, upload=233159]: Segment reached optimal threshold of 13 pieces. Dec 16 17:28:45.517288747 (265h59m45.388371812s) [noise=true , address=77.90.31.10:4520 , piece_id=HB5OCNUMFQ6X3UMEOQDSYPC2A3VNQTR7LJYZ3AOMJS3PVSIVLHGA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 16 17:28:45.517298115 (265h59m45.38838115s) [noise=true , address=77.90.31.10:4520 , piece_id=HB5OCNUMFQ6X3UMEOQDSYPC2A3VNQTR7LJYZ3AOMJS3PVSIVLHGA, seg_pos={0 0}, upload=233159]: Long tail cancellation due to optimal threshold Dec 16 17:28:45.517440477 (265h59m45.388523542s) [noise=true , address=45.91.116.5:4300 , piece_id=2C67D3NKKHRBO3J26M3PTZTQLN7DURXY2KZG5A2YKAXCMQRIWJNQ, seg_pos={0 0}, upload=233159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 16 17:28:45.517463720 (265h59m45.388546775s) [noise=true , address=45.91.116.5:4300 , piece_id=2C67D3NKKHRBO3J26M3PTZTQLN7DURXY2KZG5A2YKAXCMQRIWJNQ, seg_pos={0 0}, upload=233159]: Long tail cancellation due to optimal threshold Dec 16 17:28:45.517602394 (265h59m45.38868544s) [noise=true , address=77.90.31.11:4556 , piece_id=7V22BCWLKH422EKA3DJYUJPCAQ5LSTQRY5KB67VABMYWCXZI2MQA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 16 17:28:45.517612263 (265h59m45.388695298s) [noise=true , address=77.90.31.11:4556 , piece_id=7V22BCWLKH422EKA3DJYUJPCAQ5LSTQRY5KB67VABMYWCXZI2MQA, seg_pos={0 0}, upload=233159]: Long tail cancellation due to optimal threshold Dec 16 17:28:45.517640707 (265h59m45.388723752s) [seg_pos={0 0}, upload=233159]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:28:45.517730238 (265h59m45.388813283s) [upload=233159]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:28:45.517736550 (265h59m45.388819595s) [upload=233159]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:28:45.806056282 (265h59m45.677139317s) [upload=233159]: Done uploading segments... Dec 16 17:28:45.806074638 (265h59m45.677157673s) [upload=233159]: Upload finished. err: <nil> Duration: 5.856592986s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.420605295s trace: https://jaeger.dev.storj.io/trace/56c5b90ed930ab07 FAIL (2025-12-16 17:31:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.420633898s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [X....X................XX....X...................XX..........] duration: 3.493273264s trace: https://jaeger.dev.storj.io/trace/2d7e37a5218d47ae FAIL (2025-12-16 17:31:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.493302989s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X......X...] duration: 5.888675116s trace: https://jaeger.dev.storj.io/trace/1693cf00b42bb85 FAIL (2025-12-16 17:30:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: MVLHGEWQOWMJ6PZ4Y6EA7ALQFHGWGJFAGTPBJZLEKIY3PZP4W5YQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: VROU556GWHXECJ24OIAZQN7EB37RG4YY65EE3MPHDUZWOH3QBSEA) context deadline exceeded ======LOGS====== Dec 16 17:28:42.970569461 (265h59m42.841652506s) [upload=233159]: Starting upload Dec 16 17:28:42.970742811 (265h59m42.841825856s) [upload=233159]: Done starting upload Dec 16 17:28:42.970820098 (265h59m42.841903133s) [upload=233159]: Uploading segments... Dec 16 17:28:42.971016382 (265h59m42.842099417s) [upload=233159]: Got next segment. Inline: false Dec 16 17:28:42.971071286 (265h59m42.842154321s) [upload=233159]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:28:42.971076878 (265h59m42.842159923s) [upload=233159]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:28:43.215333084 (265h59m43.086416119s) [seg_pos={0 0}, upload=233159]: Begin upload segment... Dec 16 17:28:43.215388780 (265h59m43.086471816s) [seg_pos={0 0}, upload=233159]: Done begin upload segment. Dec 16 17:28:43.215416844 (265h59m43.086499879s) [noise=true , address=145.79.227.6:4258 , piece_id=TTPW2CX4XVWMJZV4J6JW4MRSRIGLW7Y6PEXCEYIDUNIOLM3AOYPA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.215505913 (265h59m43.086588958s) [noise=true , address=45.91.116.5:4300 , piece_id=2C67D3NKKHRBO3J26M3PTZTQLN7DURXY2KZG5A2YKAXCMQRIWJNQ, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.216114753 (265h59m43.087197798s) [noise=true , address=145.79.227.40:4049 , piece_id=SXWS3NSWYZVVXF2GWOOBKTPYRZQI4PICQJA2XD7NPAHCQGKZT6IA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.216423660 (265h59m43.087506695s) [noise=true , address=145.79.227.42:4121 , piece_id=LXGGY5STJ5UOEPYRNESNFZZ55UZY5DNYNV4PCGXSIK22KDCWUHGA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.216904276 (265h59m43.087987321s) [noise=true , address=77.90.31.11:4556 , piece_id=7V22BCWLKH422EKA3DJYUJPCAQ5LSTQRY5KB67VABMYWCXZI2MQA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.217406693 (265h59m43.088489738s) [noise=true , address=145.79.227.39:4015 , piece_id=JE2O7EODSUOO6GG4QDGR7RGQNOWY4ZCQNQ2T6RK7K26X667RMI2A, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.217976387 (265h59m43.089059422s) [noise=true , address=77.90.31.14:4664 , piece_id=VROU556GWHXECJ24OIAZQN7EB37RG4YY65EE3MPHDUZWOH3QBSEA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.219456156 (265h59m43.090539191s) [noise=true , address=145.79.227.8:4330 , piece_id=M44U54AVO6GBJA7P2ZKQGT4MJFQG7EM4CMNHFP6AMLNOV3APKVEA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.219657959 (265h59m43.090740994s) [noise=true , address=77.90.31.13:4629 , piece_id=MVLHGEWQOWMJ6PZ4Y6EA7ALQFHGWGJFAGTPBJZLEKIY3PZP4W5YQ, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.220950371 (265h59m43.092033406s) [noise=true , address=77.90.31.10:4520 , piece_id=HB5OCNUMFQ6X3UMEOQDSYPC2A3VNQTR7LJYZ3AOMJS3PVSIVLHGA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.221108251 (265h59m43.092191286s) [noise=true , address=145.79.227.7:4293 , piece_id=NANZ2FGRTALFYHJSFU5UWULLNIRMK4OGHAMO3GXH4PDRCAHUPPRA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.222409148 (265h59m43.093492183s) [noise=true , address=145.79.227.26:4655 , piece_id=D4BD3GGRPY7KRQXEJGEPEROBL6UTWE4ME36CHZM3UWF7C6KXSV5Q, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.224018112 (265h59m43.095101147s) [noise=true , address=145.79.227.50:4375 , piece_id=63ZC5AKNL6LNH7OIYODD7P5QXC7H6MVQX2IIEKUQV4BAFJMULAFA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.228049598 (265h59m43.099132673s) [noise=true , address=145.79.227.27:4689 , piece_id=ZSB47AEORTP64FGGNZG6LVCE2JPJYYXCZLL6LVVOGSNBSNTCFDHA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.230997011 (265h59m43.102080046s) [noise=true , address=145.79.227.32:4834 , piece_id=XTSKBNNSNCRT2I3ZFGNS7FNWUPDTHTVRJRRX4MSM4PYO7Y7Q3F4Q, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.233277594 (265h59m43.104360639s) [noise=true , address=77.90.31.19:4842 , piece_id=7UDB5VJP35YSCK6BZ7SB2IQC4QR7GDYXWTBDS2IAULL4C63VNHTQ, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:44.922178743 (265h59m44.793261788s) [upload=233159]: Next returned nil segment Dec 16 17:28:44.922194903 (265h59m44.793277948s) [upload=233159]: Waiting for error group managing segments... Dec 16 17:28:45.208110307 (265h59m45.079193352s) [noise=true , address=145.79.227.39:4015 , piece_id=JE2O7EODSUOO6GG4QDGR7RGQNOWY4ZCQNQ2T6RK7K26X667RMI2A, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.223090081 (265h59m45.094173116s) [noise=true , address=145.79.227.6:4258 , piece_id=TTPW2CX4XVWMJZV4J6JW4MRSRIGLW7Y6PEXCEYIDUNIOLM3AOYPA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.223193477 (265h59m45.094276513s) [noise=true , address=145.79.227.50:4375 , piece_id=63ZC5AKNL6LNH7OIYODD7P5QXC7H6MVQX2IIEKUQV4BAFJMULAFA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.235649686 (265h59m45.106732731s) [noise=true , address=145.79.227.8:4330 , piece_id=M44U54AVO6GBJA7P2ZKQGT4MJFQG7EM4CMNHFP6AMLNOV3APKVEA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.250515143 (265h59m45.121598178s) [noise=true , address=145.79.227.32:4834 , piece_id=XTSKBNNSNCRT2I3ZFGNS7FNWUPDTHTVRJRRX4MSM4PYO7Y7Q3F4Q, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.257911507 (265h59m45.128994542s) [noise=true , address=145.79.227.42:4121 , piece_id=LXGGY5STJ5UOEPYRNESNFZZ55UZY5DNYNV4PCGXSIK22KDCWUHGA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.258894539 (265h59m45.129977574s) [noise=true , address=145.79.227.40:4049 , piece_id=SXWS3NSWYZVVXF2GWOOBKTPYRZQI4PICQJA2XD7NPAHCQGKZT6IA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.286276720 (265h59m45.157359765s) [noise=true , address=145.79.227.7:4293 , piece_id=NANZ2FGRTALFYHJSFU5UWULLNIRMK4OGHAMO3GXH4PDRCAHUPPRA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.289714986 (265h59m45.160798031s) [noise=true , address=145.79.227.26:4655 , piece_id=D4BD3GGRPY7KRQXEJGEPEROBL6UTWE4ME36CHZM3UWF7C6KXSV5Q, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.300709662 (265h59m45.171792707s) [noise=true , address=145.79.227.27:4689 , piece_id=ZSB47AEORTP64FGGNZG6LVCE2JPJYYXCZLL6LVVOGSNBSNTCFDHA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.404659822 (265h59m45.275742857s) [noise=true , address=77.90.31.13:4629 , piece_id=MVLHGEWQOWMJ6PZ4Y6EA7ALQFHGWGJFAGTPBJZLEKIY3PZP4W5YQ, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.478316668 (265h59m45.349399713s) [noise=true , address=77.90.31.19:4842 , piece_id=7UDB5VJP35YSCK6BZ7SB2IQC4QR7GDYXWTBDS2IAULL4C63VNHTQ, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.516995660 (265h59m45.388078705s) [noise=true , address=77.90.31.14:4664 , piece_id=VROU556GWHXECJ24OIAZQN7EB37RG4YY65EE3MPHDUZWOH3QBSEA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.517022651 (265h59m45.388105686s) [seg_pos={0 0}, upload=233159]: Segment reached optimal threshold of 13 pieces. Dec 16 17:28:45.517288747 (265h59m45.388371812s) [noise=true , address=77.90.31.10:4520 , piece_id=HB5OCNUMFQ6X3UMEOQDSYPC2A3VNQTR7LJYZ3AOMJS3PVSIVLHGA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 16 17:28:45.517298115 (265h59m45.38838115s) [noise=true , address=77.90.31.10:4520 , piece_id=HB5OCNUMFQ6X3UMEOQDSYPC2A3VNQTR7LJYZ3AOMJS3PVSIVLHGA, seg_pos={0 0}, upload=233159]: Long tail cancellation due to optimal threshold Dec 16 17:28:45.517440477 (265h59m45.388523542s) [noise=true , address=45.91.116.5:4300 , piece_id=2C67D3NKKHRBO3J26M3PTZTQLN7DURXY2KZG5A2YKAXCMQRIWJNQ, seg_pos={0 0}, upload=233159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 16 17:28:45.517463720 (265h59m45.388546775s) [noise=true , address=45.91.116.5:4300 , piece_id=2C67D3NKKHRBO3J26M3PTZTQLN7DURXY2KZG5A2YKAXCMQRIWJNQ, seg_pos={0 0}, upload=233159]: Long tail cancellation due to optimal threshold Dec 16 17:28:45.517602394 (265h59m45.38868544s) [noise=true , address=77.90.31.11:4556 , piece_id=7V22BCWLKH422EKA3DJYUJPCAQ5LSTQRY5KB67VABMYWCXZI2MQA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 16 17:28:45.517612263 (265h59m45.388695298s) [noise=true , address=77.90.31.11:4556 , piece_id=7V22BCWLKH422EKA3DJYUJPCAQ5LSTQRY5KB67VABMYWCXZI2MQA, seg_pos={0 0}, upload=233159]: Long tail cancellation due to optimal threshold Dec 16 17:28:45.517640707 (265h59m45.388723752s) [seg_pos={0 0}, upload=233159]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:28:45.517730238 (265h59m45.388813283s) [upload=233159]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:28:45.517736550 (265h59m45.388819595s) [upload=233159]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:28:45.806056282 (265h59m45.677139317s) [upload=233159]: Done uploading segments... Dec 16 17:28:45.806074638 (265h59m45.677157673s) [upload=233159]: Upload finished. err: <nil> Duration: 5.888695755s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.X....X................XX....X...................XX.........] duration: 4.510433859s trace: https://jaeger.dev.storj.io/trace/2d7e37a5218d47ae FAIL (2025-12-16 17:30:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.510461902s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 3.00231392s trace: https://jaeger.dev.storj.io/trace/56c5b90ed930ab07 FAIL (2025-12-16 17:30:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.002338647s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X......X..] duration: 6.102235812s trace: https://jaeger.dev.storj.io/trace/1693cf00b42bb85 FAIL (2025-12-16 17:30:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: MVLHGEWQOWMJ6PZ4Y6EA7ALQFHGWGJFAGTPBJZLEKIY3PZP4W5YQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: VROU556GWHXECJ24OIAZQN7EB37RG4YY65EE3MPHDUZWOH3QBSEA) context deadline exceeded ======LOGS====== Dec 16 17:28:42.970569461 (265h59m42.841652506s) [upload=233159]: Starting upload Dec 16 17:28:42.970742811 (265h59m42.841825856s) [upload=233159]: Done starting upload Dec 16 17:28:42.970820098 (265h59m42.841903133s) [upload=233159]: Uploading segments... Dec 16 17:28:42.971016382 (265h59m42.842099417s) [upload=233159]: Got next segment. Inline: false Dec 16 17:28:42.971071286 (265h59m42.842154321s) [upload=233159]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:28:42.971076878 (265h59m42.842159923s) [upload=233159]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:28:43.215333084 (265h59m43.086416119s) [seg_pos={0 0}, upload=233159]: Begin upload segment... Dec 16 17:28:43.215388780 (265h59m43.086471816s) [seg_pos={0 0}, upload=233159]: Done begin upload segment. Dec 16 17:28:43.215416844 (265h59m43.086499879s) [noise=true , address=145.79.227.6:4258 , piece_id=TTPW2CX4XVWMJZV4J6JW4MRSRIGLW7Y6PEXCEYIDUNIOLM3AOYPA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.215505913 (265h59m43.086588958s) [noise=true , address=45.91.116.5:4300 , piece_id=2C67D3NKKHRBO3J26M3PTZTQLN7DURXY2KZG5A2YKAXCMQRIWJNQ, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.216114753 (265h59m43.087197798s) [noise=true , address=145.79.227.40:4049 , piece_id=SXWS3NSWYZVVXF2GWOOBKTPYRZQI4PICQJA2XD7NPAHCQGKZT6IA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.216423660 (265h59m43.087506695s) [noise=true , address=145.79.227.42:4121 , piece_id=LXGGY5STJ5UOEPYRNESNFZZ55UZY5DNYNV4PCGXSIK22KDCWUHGA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.216904276 (265h59m43.087987321s) [noise=true , address=77.90.31.11:4556 , piece_id=7V22BCWLKH422EKA3DJYUJPCAQ5LSTQRY5KB67VABMYWCXZI2MQA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.217406693 (265h59m43.088489738s) [noise=true , address=145.79.227.39:4015 , piece_id=JE2O7EODSUOO6GG4QDGR7RGQNOWY4ZCQNQ2T6RK7K26X667RMI2A, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.217976387 (265h59m43.089059422s) [noise=true , address=77.90.31.14:4664 , piece_id=VROU556GWHXECJ24OIAZQN7EB37RG4YY65EE3MPHDUZWOH3QBSEA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.219456156 (265h59m43.090539191s) [noise=true , address=145.79.227.8:4330 , piece_id=M44U54AVO6GBJA7P2ZKQGT4MJFQG7EM4CMNHFP6AMLNOV3APKVEA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.219657959 (265h59m43.090740994s) [noise=true , address=77.90.31.13:4629 , piece_id=MVLHGEWQOWMJ6PZ4Y6EA7ALQFHGWGJFAGTPBJZLEKIY3PZP4W5YQ, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.220950371 (265h59m43.092033406s) [noise=true , address=77.90.31.10:4520 , piece_id=HB5OCNUMFQ6X3UMEOQDSYPC2A3VNQTR7LJYZ3AOMJS3PVSIVLHGA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.221108251 (265h59m43.092191286s) [noise=true , address=145.79.227.7:4293 , piece_id=NANZ2FGRTALFYHJSFU5UWULLNIRMK4OGHAMO3GXH4PDRCAHUPPRA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.222409148 (265h59m43.093492183s) [noise=true , address=145.79.227.26:4655 , piece_id=D4BD3GGRPY7KRQXEJGEPEROBL6UTWE4ME36CHZM3UWF7C6KXSV5Q, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.224018112 (265h59m43.095101147s) [noise=true , address=145.79.227.50:4375 , piece_id=63ZC5AKNL6LNH7OIYODD7P5QXC7H6MVQX2IIEKUQV4BAFJMULAFA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.228049598 (265h59m43.099132673s) [noise=true , address=145.79.227.27:4689 , piece_id=ZSB47AEORTP64FGGNZG6LVCE2JPJYYXCZLL6LVVOGSNBSNTCFDHA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.230997011 (265h59m43.102080046s) [noise=true , address=145.79.227.32:4834 , piece_id=XTSKBNNSNCRT2I3ZFGNS7FNWUPDTHTVRJRRX4MSM4PYO7Y7Q3F4Q, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.233277594 (265h59m43.104360639s) [noise=true , address=77.90.31.19:4842 , piece_id=7UDB5VJP35YSCK6BZ7SB2IQC4QR7GDYXWTBDS2IAULL4C63VNHTQ, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:44.922178743 (265h59m44.793261788s) [upload=233159]: Next returned nil segment Dec 16 17:28:44.922194903 (265h59m44.793277948s) [upload=233159]: Waiting for error group managing segments... Dec 16 17:28:45.208110307 (265h59m45.079193352s) [noise=true , address=145.79.227.39:4015 , piece_id=JE2O7EODSUOO6GG4QDGR7RGQNOWY4ZCQNQ2T6RK7K26X667RMI2A, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.223090081 (265h59m45.094173116s) [noise=true , address=145.79.227.6:4258 , piece_id=TTPW2CX4XVWMJZV4J6JW4MRSRIGLW7Y6PEXCEYIDUNIOLM3AOYPA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.223193477 (265h59m45.094276513s) [noise=true , address=145.79.227.50:4375 , piece_id=63ZC5AKNL6LNH7OIYODD7P5QXC7H6MVQX2IIEKUQV4BAFJMULAFA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.235649686 (265h59m45.106732731s) [noise=true , address=145.79.227.8:4330 , piece_id=M44U54AVO6GBJA7P2ZKQGT4MJFQG7EM4CMNHFP6AMLNOV3APKVEA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.250515143 (265h59m45.121598178s) [noise=true , address=145.79.227.32:4834 , piece_id=XTSKBNNSNCRT2I3ZFGNS7FNWUPDTHTVRJRRX4MSM4PYO7Y7Q3F4Q, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.257911507 (265h59m45.128994542s) [noise=true , address=145.79.227.42:4121 , piece_id=LXGGY5STJ5UOEPYRNESNFZZ55UZY5DNYNV4PCGXSIK22KDCWUHGA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.258894539 (265h59m45.129977574s) [noise=true , address=145.79.227.40:4049 , piece_id=SXWS3NSWYZVVXF2GWOOBKTPYRZQI4PICQJA2XD7NPAHCQGKZT6IA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.286276720 (265h59m45.157359765s) [noise=true , address=145.79.227.7:4293 , piece_id=NANZ2FGRTALFYHJSFU5UWULLNIRMK4OGHAMO3GXH4PDRCAHUPPRA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.289714986 (265h59m45.160798031s) [noise=true , address=145.79.227.26:4655 , piece_id=D4BD3GGRPY7KRQXEJGEPEROBL6UTWE4ME36CHZM3UWF7C6KXSV5Q, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.300709662 (265h59m45.171792707s) [noise=true , address=145.79.227.27:4689 , piece_id=ZSB47AEORTP64FGGNZG6LVCE2JPJYYXCZLL6LVVOGSNBSNTCFDHA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.404659822 (265h59m45.275742857s) [noise=true , address=77.90.31.13:4629 , piece_id=MVLHGEWQOWMJ6PZ4Y6EA7ALQFHGWGJFAGTPBJZLEKIY3PZP4W5YQ, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.478316668 (265h59m45.349399713s) [noise=true , address=77.90.31.19:4842 , piece_id=7UDB5VJP35YSCK6BZ7SB2IQC4QR7GDYXWTBDS2IAULL4C63VNHTQ, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.516995660 (265h59m45.388078705s) [noise=true , address=77.90.31.14:4664 , piece_id=VROU556GWHXECJ24OIAZQN7EB37RG4YY65EE3MPHDUZWOH3QBSEA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.517022651 (265h59m45.388105686s) [seg_pos={0 0}, upload=233159]: Segment reached optimal threshold of 13 pieces. Dec 16 17:28:45.517288747 (265h59m45.388371812s) [noise=true , address=77.90.31.10:4520 , piece_id=HB5OCNUMFQ6X3UMEOQDSYPC2A3VNQTR7LJYZ3AOMJS3PVSIVLHGA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 16 17:28:45.517298115 (265h59m45.38838115s) [noise=true , address=77.90.31.10:4520 , piece_id=HB5OCNUMFQ6X3UMEOQDSYPC2A3VNQTR7LJYZ3AOMJS3PVSIVLHGA, seg_pos={0 0}, upload=233159]: Long tail cancellation due to optimal threshold Dec 16 17:28:45.517440477 (265h59m45.388523542s) [noise=true , address=45.91.116.5:4300 , piece_id=2C67D3NKKHRBO3J26M3PTZTQLN7DURXY2KZG5A2YKAXCMQRIWJNQ, seg_pos={0 0}, upload=233159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 16 17:28:45.517463720 (265h59m45.388546775s) [noise=true , address=45.91.116.5:4300 , piece_id=2C67D3NKKHRBO3J26M3PTZTQLN7DURXY2KZG5A2YKAXCMQRIWJNQ, seg_pos={0 0}, upload=233159]: Long tail cancellation due to optimal threshold Dec 16 17:28:45.517602394 (265h59m45.38868544s) [noise=true , address=77.90.31.11:4556 , piece_id=7V22BCWLKH422EKA3DJYUJPCAQ5LSTQRY5KB67VABMYWCXZI2MQA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 16 17:28:45.517612263 (265h59m45.388695298s) [noise=true , address=77.90.31.11:4556 , piece_id=7V22BCWLKH422EKA3DJYUJPCAQ5LSTQRY5KB67VABMYWCXZI2MQA, seg_pos={0 0}, upload=233159]: Long tail cancellation due to optimal threshold Dec 16 17:28:45.517640707 (265h59m45.388723752s) [seg_pos={0 0}, upload=233159]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:28:45.517730238 (265h59m45.388813283s) [upload=233159]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:28:45.517736550 (265h59m45.388819595s) [upload=233159]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:28:45.806056282 (265h59m45.677139317s) [upload=233159]: Done uploading segments... Dec 16 17:28:45.806074638 (265h59m45.677157673s) [upload=233159]: Upload finished. err: <nil> Duration: 6.102255921s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..X....X................XX....X...................XX........] duration: 3.851863023s trace: https://jaeger.dev.storj.io/trace/2d7e37a5218d47ae FAIL (2025-12-16 17:30:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.851894823s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.306097181s trace: https://jaeger.dev.storj.io/trace/56c5b90ed930ab07 FAIL (2025-12-16 17:29:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.306126185s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X......X.] duration: 6.009053697s trace: https://jaeger.dev.storj.io/trace/1693cf00b42bb85 FAIL (2025-12-16 17:29:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: MVLHGEWQOWMJ6PZ4Y6EA7ALQFHGWGJFAGTPBJZLEKIY3PZP4W5YQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: VROU556GWHXECJ24OIAZQN7EB37RG4YY65EE3MPHDUZWOH3QBSEA) context deadline exceeded ======LOGS====== Dec 16 17:28:42.970569461 (265h59m42.841652506s) [upload=233159]: Starting upload Dec 16 17:28:42.970742811 (265h59m42.841825856s) [upload=233159]: Done starting upload Dec 16 17:28:42.970820098 (265h59m42.841903133s) [upload=233159]: Uploading segments... Dec 16 17:28:42.971016382 (265h59m42.842099417s) [upload=233159]: Got next segment. Inline: false Dec 16 17:28:42.971071286 (265h59m42.842154321s) [upload=233159]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:28:42.971076878 (265h59m42.842159923s) [upload=233159]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:28:43.215333084 (265h59m43.086416119s) [seg_pos={0 0}, upload=233159]: Begin upload segment... Dec 16 17:28:43.215388780 (265h59m43.086471816s) [seg_pos={0 0}, upload=233159]: Done begin upload segment. Dec 16 17:28:43.215416844 (265h59m43.086499879s) [noise=true , address=145.79.227.6:4258 , piece_id=TTPW2CX4XVWMJZV4J6JW4MRSRIGLW7Y6PEXCEYIDUNIOLM3AOYPA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.215505913 (265h59m43.086588958s) [noise=true , address=45.91.116.5:4300 , piece_id=2C67D3NKKHRBO3J26M3PTZTQLN7DURXY2KZG5A2YKAXCMQRIWJNQ, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.216114753 (265h59m43.087197798s) [noise=true , address=145.79.227.40:4049 , piece_id=SXWS3NSWYZVVXF2GWOOBKTPYRZQI4PICQJA2XD7NPAHCQGKZT6IA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.216423660 (265h59m43.087506695s) [noise=true , address=145.79.227.42:4121 , piece_id=LXGGY5STJ5UOEPYRNESNFZZ55UZY5DNYNV4PCGXSIK22KDCWUHGA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.216904276 (265h59m43.087987321s) [noise=true , address=77.90.31.11:4556 , piece_id=7V22BCWLKH422EKA3DJYUJPCAQ5LSTQRY5KB67VABMYWCXZI2MQA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.217406693 (265h59m43.088489738s) [noise=true , address=145.79.227.39:4015 , piece_id=JE2O7EODSUOO6GG4QDGR7RGQNOWY4ZCQNQ2T6RK7K26X667RMI2A, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.217976387 (265h59m43.089059422s) [noise=true , address=77.90.31.14:4664 , piece_id=VROU556GWHXECJ24OIAZQN7EB37RG4YY65EE3MPHDUZWOH3QBSEA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.219456156 (265h59m43.090539191s) [noise=true , address=145.79.227.8:4330 , piece_id=M44U54AVO6GBJA7P2ZKQGT4MJFQG7EM4CMNHFP6AMLNOV3APKVEA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.219657959 (265h59m43.090740994s) [noise=true , address=77.90.31.13:4629 , piece_id=MVLHGEWQOWMJ6PZ4Y6EA7ALQFHGWGJFAGTPBJZLEKIY3PZP4W5YQ, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.220950371 (265h59m43.092033406s) [noise=true , address=77.90.31.10:4520 , piece_id=HB5OCNUMFQ6X3UMEOQDSYPC2A3VNQTR7LJYZ3AOMJS3PVSIVLHGA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.221108251 (265h59m43.092191286s) [noise=true , address=145.79.227.7:4293 , piece_id=NANZ2FGRTALFYHJSFU5UWULLNIRMK4OGHAMO3GXH4PDRCAHUPPRA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.222409148 (265h59m43.093492183s) [noise=true , address=145.79.227.26:4655 , piece_id=D4BD3GGRPY7KRQXEJGEPEROBL6UTWE4ME36CHZM3UWF7C6KXSV5Q, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.224018112 (265h59m43.095101147s) [noise=true , address=145.79.227.50:4375 , piece_id=63ZC5AKNL6LNH7OIYODD7P5QXC7H6MVQX2IIEKUQV4BAFJMULAFA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.228049598 (265h59m43.099132673s) [noise=true , address=145.79.227.27:4689 , piece_id=ZSB47AEORTP64FGGNZG6LVCE2JPJYYXCZLL6LVVOGSNBSNTCFDHA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.230997011 (265h59m43.102080046s) [noise=true , address=145.79.227.32:4834 , piece_id=XTSKBNNSNCRT2I3ZFGNS7FNWUPDTHTVRJRRX4MSM4PYO7Y7Q3F4Q, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.233277594 (265h59m43.104360639s) [noise=true , address=77.90.31.19:4842 , piece_id=7UDB5VJP35YSCK6BZ7SB2IQC4QR7GDYXWTBDS2IAULL4C63VNHTQ, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:44.922178743 (265h59m44.793261788s) [upload=233159]: Next returned nil segment Dec 16 17:28:44.922194903 (265h59m44.793277948s) [upload=233159]: Waiting for error group managing segments... Dec 16 17:28:45.208110307 (265h59m45.079193352s) [noise=true , address=145.79.227.39:4015 , piece_id=JE2O7EODSUOO6GG4QDGR7RGQNOWY4ZCQNQ2T6RK7K26X667RMI2A, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.223090081 (265h59m45.094173116s) [noise=true , address=145.79.227.6:4258 , piece_id=TTPW2CX4XVWMJZV4J6JW4MRSRIGLW7Y6PEXCEYIDUNIOLM3AOYPA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.223193477 (265h59m45.094276513s) [noise=true , address=145.79.227.50:4375 , piece_id=63ZC5AKNL6LNH7OIYODD7P5QXC7H6MVQX2IIEKUQV4BAFJMULAFA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.235649686 (265h59m45.106732731s) [noise=true , address=145.79.227.8:4330 , piece_id=M44U54AVO6GBJA7P2ZKQGT4MJFQG7EM4CMNHFP6AMLNOV3APKVEA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.250515143 (265h59m45.121598178s) [noise=true , address=145.79.227.32:4834 , piece_id=XTSKBNNSNCRT2I3ZFGNS7FNWUPDTHTVRJRRX4MSM4PYO7Y7Q3F4Q, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.257911507 (265h59m45.128994542s) [noise=true , address=145.79.227.42:4121 , piece_id=LXGGY5STJ5UOEPYRNESNFZZ55UZY5DNYNV4PCGXSIK22KDCWUHGA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.258894539 (265h59m45.129977574s) [noise=true , address=145.79.227.40:4049 , piece_id=SXWS3NSWYZVVXF2GWOOBKTPYRZQI4PICQJA2XD7NPAHCQGKZT6IA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.286276720 (265h59m45.157359765s) [noise=true , address=145.79.227.7:4293 , piece_id=NANZ2FGRTALFYHJSFU5UWULLNIRMK4OGHAMO3GXH4PDRCAHUPPRA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.289714986 (265h59m45.160798031s) [noise=true , address=145.79.227.26:4655 , piece_id=D4BD3GGRPY7KRQXEJGEPEROBL6UTWE4ME36CHZM3UWF7C6KXSV5Q, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.300709662 (265h59m45.171792707s) [noise=true , address=145.79.227.27:4689 , piece_id=ZSB47AEORTP64FGGNZG6LVCE2JPJYYXCZLL6LVVOGSNBSNTCFDHA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.404659822 (265h59m45.275742857s) [noise=true , address=77.90.31.13:4629 , piece_id=MVLHGEWQOWMJ6PZ4Y6EA7ALQFHGWGJFAGTPBJZLEKIY3PZP4W5YQ, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.478316668 (265h59m45.349399713s) [noise=true , address=77.90.31.19:4842 , piece_id=7UDB5VJP35YSCK6BZ7SB2IQC4QR7GDYXWTBDS2IAULL4C63VNHTQ, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.516995660 (265h59m45.388078705s) [noise=true , address=77.90.31.14:4664 , piece_id=VROU556GWHXECJ24OIAZQN7EB37RG4YY65EE3MPHDUZWOH3QBSEA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.517022651 (265h59m45.388105686s) [seg_pos={0 0}, upload=233159]: Segment reached optimal threshold of 13 pieces. Dec 16 17:28:45.517288747 (265h59m45.388371812s) [noise=true , address=77.90.31.10:4520 , piece_id=HB5OCNUMFQ6X3UMEOQDSYPC2A3VNQTR7LJYZ3AOMJS3PVSIVLHGA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 16 17:28:45.517298115 (265h59m45.38838115s) [noise=true , address=77.90.31.10:4520 , piece_id=HB5OCNUMFQ6X3UMEOQDSYPC2A3VNQTR7LJYZ3AOMJS3PVSIVLHGA, seg_pos={0 0}, upload=233159]: Long tail cancellation due to optimal threshold Dec 16 17:28:45.517440477 (265h59m45.388523542s) [noise=true , address=45.91.116.5:4300 , piece_id=2C67D3NKKHRBO3J26M3PTZTQLN7DURXY2KZG5A2YKAXCMQRIWJNQ, seg_pos={0 0}, upload=233159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 16 17:28:45.517463720 (265h59m45.388546775s) [noise=true , address=45.91.116.5:4300 , piece_id=2C67D3NKKHRBO3J26M3PTZTQLN7DURXY2KZG5A2YKAXCMQRIWJNQ, seg_pos={0 0}, upload=233159]: Long tail cancellation due to optimal threshold Dec 16 17:28:45.517602394 (265h59m45.38868544s) [noise=true , address=77.90.31.11:4556 , piece_id=7V22BCWLKH422EKA3DJYUJPCAQ5LSTQRY5KB67VABMYWCXZI2MQA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 16 17:28:45.517612263 (265h59m45.388695298s) [noise=true , address=77.90.31.11:4556 , piece_id=7V22BCWLKH422EKA3DJYUJPCAQ5LSTQRY5KB67VABMYWCXZI2MQA, seg_pos={0 0}, upload=233159]: Long tail cancellation due to optimal threshold Dec 16 17:28:45.517640707 (265h59m45.388723752s) [seg_pos={0 0}, upload=233159]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:28:45.517730238 (265h59m45.388813283s) [upload=233159]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:28:45.517736550 (265h59m45.388819595s) [upload=233159]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:28:45.806056282 (265h59m45.677139317s) [upload=233159]: Done uploading segments... Dec 16 17:28:45.806074638 (265h59m45.677157673s) [upload=233159]: Upload finished. err: <nil> Duration: 6.009077672s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X...................................................X......X] duration: 30.007183207s trace: https://jaeger.dev.storj.io/trace/1693cf00b42bb85 FAIL (2025-12-16 17:29:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: MVLHGEWQOWMJ6PZ4Y6EA7ALQFHGWGJFAGTPBJZLEKIY3PZP4W5YQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: VROU556GWHXECJ24OIAZQN7EB37RG4YY65EE3MPHDUZWOH3QBSEA) context deadline exceeded ======LOGS====== Dec 16 17:28:42.970569461 (265h59m42.841652506s) [upload=233159]: Starting upload Dec 16 17:28:42.970742811 (265h59m42.841825856s) [upload=233159]: Done starting upload Dec 16 17:28:42.970820098 (265h59m42.841903133s) [upload=233159]: Uploading segments... Dec 16 17:28:42.971016382 (265h59m42.842099417s) [upload=233159]: Got next segment. Inline: false Dec 16 17:28:42.971071286 (265h59m42.842154321s) [upload=233159]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:28:42.971076878 (265h59m42.842159923s) [upload=233159]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:28:43.215333084 (265h59m43.086416119s) [seg_pos={0 0}, upload=233159]: Begin upload segment... Dec 16 17:28:43.215388780 (265h59m43.086471816s) [seg_pos={0 0}, upload=233159]: Done begin upload segment. Dec 16 17:28:43.215416844 (265h59m43.086499879s) [noise=true , address=145.79.227.6:4258 , piece_id=TTPW2CX4XVWMJZV4J6JW4MRSRIGLW7Y6PEXCEYIDUNIOLM3AOYPA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.215505913 (265h59m43.086588958s) [noise=true , address=45.91.116.5:4300 , piece_id=2C67D3NKKHRBO3J26M3PTZTQLN7DURXY2KZG5A2YKAXCMQRIWJNQ, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.216114753 (265h59m43.087197798s) [noise=true , address=145.79.227.40:4049 , piece_id=SXWS3NSWYZVVXF2GWOOBKTPYRZQI4PICQJA2XD7NPAHCQGKZT6IA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.216423660 (265h59m43.087506695s) [noise=true , address=145.79.227.42:4121 , piece_id=LXGGY5STJ5UOEPYRNESNFZZ55UZY5DNYNV4PCGXSIK22KDCWUHGA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.216904276 (265h59m43.087987321s) [noise=true , address=77.90.31.11:4556 , piece_id=7V22BCWLKH422EKA3DJYUJPCAQ5LSTQRY5KB67VABMYWCXZI2MQA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.217406693 (265h59m43.088489738s) [noise=true , address=145.79.227.39:4015 , piece_id=JE2O7EODSUOO6GG4QDGR7RGQNOWY4ZCQNQ2T6RK7K26X667RMI2A, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.217976387 (265h59m43.089059422s) [noise=true , address=77.90.31.14:4664 , piece_id=VROU556GWHXECJ24OIAZQN7EB37RG4YY65EE3MPHDUZWOH3QBSEA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.219456156 (265h59m43.090539191s) [noise=true , address=145.79.227.8:4330 , piece_id=M44U54AVO6GBJA7P2ZKQGT4MJFQG7EM4CMNHFP6AMLNOV3APKVEA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.219657959 (265h59m43.090740994s) [noise=true , address=77.90.31.13:4629 , piece_id=MVLHGEWQOWMJ6PZ4Y6EA7ALQFHGWGJFAGTPBJZLEKIY3PZP4W5YQ, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.220950371 (265h59m43.092033406s) [noise=true , address=77.90.31.10:4520 , piece_id=HB5OCNUMFQ6X3UMEOQDSYPC2A3VNQTR7LJYZ3AOMJS3PVSIVLHGA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.221108251 (265h59m43.092191286s) [noise=true , address=145.79.227.7:4293 , piece_id=NANZ2FGRTALFYHJSFU5UWULLNIRMK4OGHAMO3GXH4PDRCAHUPPRA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.222409148 (265h59m43.093492183s) [noise=true , address=145.79.227.26:4655 , piece_id=D4BD3GGRPY7KRQXEJGEPEROBL6UTWE4ME36CHZM3UWF7C6KXSV5Q, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.224018112 (265h59m43.095101147s) [noise=true , address=145.79.227.50:4375 , piece_id=63ZC5AKNL6LNH7OIYODD7P5QXC7H6MVQX2IIEKUQV4BAFJMULAFA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.228049598 (265h59m43.099132673s) [noise=true , address=145.79.227.27:4689 , piece_id=ZSB47AEORTP64FGGNZG6LVCE2JPJYYXCZLL6LVVOGSNBSNTCFDHA, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.230997011 (265h59m43.102080046s) [noise=true , address=145.79.227.32:4834 , piece_id=XTSKBNNSNCRT2I3ZFGNS7FNWUPDTHTVRJRRX4MSM4PYO7Y7Q3F4Q, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:43.233277594 (265h59m43.104360639s) [noise=true , address=77.90.31.19:4842 , piece_id=7UDB5VJP35YSCK6BZ7SB2IQC4QR7GDYXWTBDS2IAULL4C63VNHTQ, seg_pos={0 0}, upload=233159]: Uploading piece... Dec 16 17:28:44.922178743 (265h59m44.793261788s) [upload=233159]: Next returned nil segment Dec 16 17:28:44.922194903 (265h59m44.793277948s) [upload=233159]: Waiting for error group managing segments... Dec 16 17:28:45.208110307 (265h59m45.079193352s) [noise=true , address=145.79.227.39:4015 , piece_id=JE2O7EODSUOO6GG4QDGR7RGQNOWY4ZCQNQ2T6RK7K26X667RMI2A, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.223090081 (265h59m45.094173116s) [noise=true , address=145.79.227.6:4258 , piece_id=TTPW2CX4XVWMJZV4J6JW4MRSRIGLW7Y6PEXCEYIDUNIOLM3AOYPA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.223193477 (265h59m45.094276513s) [noise=true , address=145.79.227.50:4375 , piece_id=63ZC5AKNL6LNH7OIYODD7P5QXC7H6MVQX2IIEKUQV4BAFJMULAFA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.235649686 (265h59m45.106732731s) [noise=true , address=145.79.227.8:4330 , piece_id=M44U54AVO6GBJA7P2ZKQGT4MJFQG7EM4CMNHFP6AMLNOV3APKVEA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.250515143 (265h59m45.121598178s) [noise=true , address=145.79.227.32:4834 , piece_id=XTSKBNNSNCRT2I3ZFGNS7FNWUPDTHTVRJRRX4MSM4PYO7Y7Q3F4Q, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.257911507 (265h59m45.128994542s) [noise=true , address=145.79.227.42:4121 , piece_id=LXGGY5STJ5UOEPYRNESNFZZ55UZY5DNYNV4PCGXSIK22KDCWUHGA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.258894539 (265h59m45.129977574s) [noise=true , address=145.79.227.40:4049 , piece_id=SXWS3NSWYZVVXF2GWOOBKTPYRZQI4PICQJA2XD7NPAHCQGKZT6IA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.286276720 (265h59m45.157359765s) [noise=true , address=145.79.227.7:4293 , piece_id=NANZ2FGRTALFYHJSFU5UWULLNIRMK4OGHAMO3GXH4PDRCAHUPPRA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.289714986 (265h59m45.160798031s) [noise=true , address=145.79.227.26:4655 , piece_id=D4BD3GGRPY7KRQXEJGEPEROBL6UTWE4ME36CHZM3UWF7C6KXSV5Q, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.300709662 (265h59m45.171792707s) [noise=true , address=145.79.227.27:4689 , piece_id=ZSB47AEORTP64FGGNZG6LVCE2JPJYYXCZLL6LVVOGSNBSNTCFDHA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.404659822 (265h59m45.275742857s) [noise=true , address=77.90.31.13:4629 , piece_id=MVLHGEWQOWMJ6PZ4Y6EA7ALQFHGWGJFAGTPBJZLEKIY3PZP4W5YQ, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.478316668 (265h59m45.349399713s) [noise=true , address=77.90.31.19:4842 , piece_id=7UDB5VJP35YSCK6BZ7SB2IQC4QR7GDYXWTBDS2IAULL4C63VNHTQ, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.516995660 (265h59m45.388078705s) [noise=true , address=77.90.31.14:4664 , piece_id=VROU556GWHXECJ24OIAZQN7EB37RG4YY65EE3MPHDUZWOH3QBSEA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: <nil> Dec 16 17:28:45.517022651 (265h59m45.388105686s) [seg_pos={0 0}, upload=233159]: Segment reached optimal threshold of 13 pieces. Dec 16 17:28:45.517288747 (265h59m45.388371812s) [noise=true , address=77.90.31.10:4520 , piece_id=HB5OCNUMFQ6X3UMEOQDSYPC2A3VNQTR7LJYZ3AOMJS3PVSIVLHGA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 16 17:28:45.517298115 (265h59m45.38838115s) [noise=true , address=77.90.31.10:4520 , piece_id=HB5OCNUMFQ6X3UMEOQDSYPC2A3VNQTR7LJYZ3AOMJS3PVSIVLHGA, seg_pos={0 0}, upload=233159]: Long tail cancellation due to optimal threshold Dec 16 17:28:45.517440477 (265h59m45.388523542s) [noise=true , address=45.91.116.5:4300 , piece_id=2C67D3NKKHRBO3J26M3PTZTQLN7DURXY2KZG5A2YKAXCMQRIWJNQ, seg_pos={0 0}, upload=233159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 16 17:28:45.517463720 (265h59m45.388546775s) [noise=true , address=45.91.116.5:4300 , piece_id=2C67D3NKKHRBO3J26M3PTZTQLN7DURXY2KZG5A2YKAXCMQRIWJNQ, seg_pos={0 0}, upload=233159]: Long tail cancellation due to optimal threshold Dec 16 17:28:45.517602394 (265h59m45.38868544s) [noise=true , address=77.90.31.11:4556 , piece_id=7V22BCWLKH422EKA3DJYUJPCAQ5LSTQRY5KB67VABMYWCXZI2MQA, seg_pos={0 0}, upload=233159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 16 17:28:45.517612263 (265h59m45.388695298s) [noise=true , address=77.90.31.11:4556 , piece_id=7V22BCWLKH422EKA3DJYUJPCAQ5LSTQRY5KB67VABMYWCXZI2MQA, seg_pos={0 0}, upload=233159]: Long tail cancellation due to optimal threshold Dec 16 17:28:45.517640707 (265h59m45.388723752s) [seg_pos={0 0}, upload=233159]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:28:45.517730238 (265h59m45.388813283s) [upload=233159]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:28:45.517736550 (265h59m45.388819595s) [upload=233159]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:28:45.806056282 (265h59m45.677139317s) [upload=233159]: Done uploading segments... Dec 16 17:28:45.806074638 (265h59m45.677157673s) [upload=233159]: Upload finished. err: <nil> Duration: 30.00721062s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [X..X....X................XX....X...................XX.......] duration: 5.001252264s trace: https://jaeger.dev.storj.io/trace/2d7e37a5218d47ae FAIL (2025-12-16 17:29:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001262794s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.X..X....X................XX....X...................XX......] duration: 5.001196058s trace: https://jaeger.dev.storj.io/trace/2d7e37a5218d47ae FAIL (2025-12-16 17:28:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00121823s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.366196339s trace: https://jaeger.dev.storj.io/trace/56c5b90ed930ab07 FAIL (2025-12-16 17:28:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.366221267s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X...................................................X......] duration: 5.776750351s trace: https://jaeger.dev.storj.io/trace/300ebb63ed4f1615 FAIL (2025-12-16 17:27:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 16 17:22:28.355769386 (265h53m28.226852431s) [upload=233070]: Starting upload Dec 16 17:22:28.355830982 (265h53m28.226914028s) [upload=233070]: Done starting upload Dec 16 17:22:28.355920163 (265h53m28.227003208s) [upload=233070]: Uploading segments... Dec 16 17:22:28.356090597 (265h53m28.227173632s) [upload=233070]: Got next segment. Inline: false Dec 16 17:22:28.356131766 (265h53m28.227214801s) [upload=233070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:22:28.356167273 (265h53m28.227250318s) [upload=233070]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:22:28.602538979 (265h53m28.473622024s) [seg_pos={0 0}, upload=233070]: Begin upload segment... Dec 16 17:22:28.602655190 (265h53m28.473738235s) [seg_pos={0 0}, upload=233070]: Done begin upload segment. Dec 16 17:22:28.602702590 (265h53m28.473785635s) [noise=true , address=77.90.31.11:4554 , piece_id=IMT5XXWJZL3LHQMWVMCIU7WPX2OGOFWQ6EH7HPOO3O4ANN6PVSIA, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.602802460 (265h53m28.473885505s) [noise=true , address=45.91.116.10:4081 , piece_id=F6X4FPZ3PYRL2V6B5BTZLL36JSDIFA2AZUMB2Q3TEBVHDOPDGBRA, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.603472637 (265h53m28.474555672s) [noise=true , address=45.91.116.13:4154 , piece_id=EWWYIQY4FLTYPK4GDOAQPT4CTSM4WZBIJB2TZILD65G5RD6LN4XA, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.603827322 (265h53m28.474910367s) [noise=true , address=45.91.116.7:4009 , piece_id=54XGXE5QJ6UZJBV67K4DBDMPXG6A72PPSE3UQ3OXHU53RFP6KZ3A, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.603896484 (265h53m28.474979519s) [noise=true , address=145.79.227.35:4481 , piece_id=R5KOPMCY42A45GH2HEAIHRUXQORFVSJEYW7YAYU23HEDH6OUBQWQ, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.604214349 (265h53m28.475297394s) [noise=true , address=145.79.227.46:4601 , piece_id=WV6ECF6MO4W65RNLWVHGCM4RC4I5E627UOTRQR6PTQ74ZJRIKD2A, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.604287829 (265h53m28.475370864s) [noise=true , address=45.91.116.11:4118 , piece_id=MXVHHSHA7CHNU7BQZVFE52JRATWAUBKFMAW662FQPPDIKANVGK2Q, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.604632165 (265h53m28.4757152s) [noise=true , address=45.91.116.14:4191 , piece_id=ELCLNIWDWE2STLW5U6FZL6OMIEMRGUSMEB4E3CTUX5V64RMXP2QQ, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.604661400 (265h53m28.475744446s) [noise=true , address=145.79.227.34:4907 , piece_id=C5GHGT5OGA7XRJIHJANVBAIF7Z6VMCDKI6WPEFTBDZPUKMIPLO5A, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.605019282 (265h53m28.476102317s) [noise=true , address=77.90.31.21:4877 , piece_id=MWYGIKFVXISWBUXVB4WEWBE3BOP4VCIOD6I3MBWBYCZEV7YCIHNQ, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.605098462 (265h53m28.476181507s) [noise=true , address=145.79.227.27:4691 , piece_id=DT5BT7KS4FTRIGYLM3OQWHULUYOZUNEP6FZ7XUBB2RMORPPUM5ZQ, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.605462636 (265h53m28.476545681s) [noise=true , address=145.79.227.52:4446 , piece_id=4RN2ZBGTULZ6VQSO4H6VEKIDEG2DLMDUW74LMUXOZGPXNBWF7GHQ, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.605494667 (265h53m28.476577712s) [noise=true , address=145.79.227.11:4732 , piece_id=3OSGIDEGOSVIMIBBTTRBMXJHGT626AC4GYDC3U2OY62SWB6FEV6A, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.605842790 (265h53m28.476925825s) [noise=true , address=45.91.116.8:4047 , piece_id=SOIENQXK3MCGBRBA6FOKCHEDEWIUMHHRUXO4FSPG2525XY4DXZCA, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.606015409 (265h53m28.477098454s) [noise=true , address=77.90.31.18:4808 , piece_id=QJT3MASV2WG5WBUSDLTETUHEL74QP4NWC5JFXVQQF5WYXY6DCOJA, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.606383579 (265h53m28.477466624s) [noise=true , address=45.91.116.5:4297 , piece_id=YT3FOYJEKBYTGDMA4Q2MKMJBLQOHQVPFZXSI322GZ7HGQDZ33ESA, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:29.753134117 (265h53m29.624217162s) [noise=true , address=77.90.31.21:4877 , piece_id=MWYGIKFVXISWBUXVB4WEWBE3BOP4VCIOD6I3MBWBYCZEV7YCIHNQ, seg_pos={0 0}, upload=233070]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 16 17:22:30.332323516 (265h53m30.203406561s) [upload=233070]: Next returned nil segment Dec 16 17:22:30.332340469 (265h53m30.203423514s) [upload=233070]: Waiting for error group managing segments... Dec 16 17:22:30.616216770 (265h53m30.487299815s) [noise=true , address=145.79.227.35:4481 , piece_id=R5KOPMCY42A45GH2HEAIHRUXQORFVSJEYW7YAYU23HEDH6OUBQWQ, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.622252243 (265h53m30.493335278s) [noise=true , address=145.79.227.52:4446 , piece_id=4RN2ZBGTULZ6VQSO4H6VEKIDEG2DLMDUW74LMUXOZGPXNBWF7GHQ, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.623047456 (265h53m30.494130491s) [noise=true , address=145.79.227.46:4601 , piece_id=WV6ECF6MO4W65RNLWVHGCM4RC4I5E627UOTRQR6PTQ74ZJRIKD2A, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.633939096 (265h53m30.505022131s) [noise=true , address=145.79.227.27:4691 , piece_id=DT5BT7KS4FTRIGYLM3OQWHULUYOZUNEP6FZ7XUBB2RMORPPUM5ZQ, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.676237018 (265h53m30.547320063s) [noise=true , address=145.79.227.34:4907 , piece_id=C5GHGT5OGA7XRJIHJANVBAIF7Z6VMCDKI6WPEFTBDZPUKMIPLO5A, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.684076116 (265h53m30.555159151s) [noise=true , address=145.79.227.11:4732 , piece_id=3OSGIDEGOSVIMIBBTTRBMXJHGT626AC4GYDC3U2OY62SWB6FEV6A, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.851675636 (265h53m30.722758681s) [noise=true , address=77.90.31.11:4554 , piece_id=IMT5XXWJZL3LHQMWVMCIU7WPX2OGOFWQ6EH7HPOO3O4ANN6PVSIA, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.923476907 (265h53m30.794559942s) [noise=true , address=45.91.116.11:4118 , piece_id=MXVHHSHA7CHNU7BQZVFE52JRATWAUBKFMAW662FQPPDIKANVGK2Q, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.927061062 (265h53m30.798144107s) [noise=true , address=45.91.116.8:4047 , piece_id=SOIENQXK3MCGBRBA6FOKCHEDEWIUMHHRUXO4FSPG2525XY4DXZCA, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.940531031 (265h53m30.811614067s) [noise=true , address=45.91.116.14:4191 , piece_id=ELCLNIWDWE2STLW5U6FZL6OMIEMRGUSMEB4E3CTUX5V64RMXP2QQ, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.956255193 (265h53m30.827338238s) [noise=true , address=45.91.116.5:4297 , piece_id=YT3FOYJEKBYTGDMA4Q2MKMJBLQOHQVPFZXSI322GZ7HGQDZ33ESA, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:31.004934157 (265h53m30.876017202s) [noise=true , address=45.91.116.7:4009 , piece_id=54XGXE5QJ6UZJBV67K4DBDMPXG6A72PPSE3UQ3OXHU53RFP6KZ3A, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:31.126466316 (265h53m30.997549361s) [noise=true , address=45.91.116.13:4154 , piece_id=EWWYIQY4FLTYPK4GDOAQPT4CTSM4WZBIJB2TZILD65G5RD6LN4XA, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:31.126490482 (265h53m30.997573517s) [seg_pos={0 0}, upload=233070]: Segment reached optimal threshold of 13 pieces. Dec 16 17:22:31.126883510 (265h53m30.997966556s) [noise=true , address=45.91.116.10:4081 , piece_id=F6X4FPZ3PYRL2V6B5BTZLL36JSDIFA2AZUMB2Q3TEBVHDOPDGBRA, seg_pos={0 0}, upload=233070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:60060->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:60060->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60060->45.91.116.10:4081: use of closed network connection Dec 16 17:22:31.126900472 (265h53m30.997983517s) [noise=true , address=45.91.116.10:4081 , piece_id=F6X4FPZ3PYRL2V6B5BTZLL36JSDIFA2AZUMB2Q3TEBVHDOPDGBRA, seg_pos={0 0}, upload=233070]: Long tail cancellation due to optimal threshold Dec 16 17:22:31.126980125 (265h53m30.99806317s) [noise=true , address=77.90.31.18:4808 , piece_id=QJT3MASV2WG5WBUSDLTETUHEL74QP4NWC5JFXVQQF5WYXY6DCOJA, seg_pos={0 0}, upload=233070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:35310->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:35310->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35310->77.90.31.18:4808: use of closed network connection Dec 16 17:22:31.126991246 (265h53m30.998074291s) [noise=true , address=77.90.31.18:4808 , piece_id=QJT3MASV2WG5WBUSDLTETUHEL74QP4NWC5JFXVQQF5WYXY6DCOJA, seg_pos={0 0}, upload=233070]: Long tail cancellation due to optimal threshold Dec 16 17:22:31.127010171 (265h53m30.998093216s) [seg_pos={0 0}, upload=233070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:22:31.127091877 (265h53m30.998174912s) [upload=233070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:22:31.127098369 (265h53m30.998181404s) [upload=233070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:22:31.423343183 (265h53m31.294426218s) [upload=233070]: Done uploading segments... Dec 16 17:22:31.423362178 (265h53m31.294445213s) [upload=233070]: Upload finished. err: <nil> Duration: 5.77677565s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [..X..X....X................XX....X...................XX.....] duration: 4.485820912s trace: https://jaeger.dev.storj.io/trace/2d7e37a5218d47ae FAIL (2025-12-16 17:27:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.485853423s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.334159185s trace: https://jaeger.dev.storj.io/trace/56c5b90ed930ab07 FAIL (2025-12-16 17:27:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.334186788s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 3.068842252s trace: https://jaeger.dev.storj.io/trace/56c5b90ed930ab07 FAIL (2025-12-16 17:26:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.068865435s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X...................................................X.....] duration: 6.025748378s trace: https://jaeger.dev.storj.io/trace/300ebb63ed4f1615 FAIL (2025-12-16 17:26:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 16 17:22:28.355769386 (265h53m28.226852431s) [upload=233070]: Starting upload Dec 16 17:22:28.355830982 (265h53m28.226914028s) [upload=233070]: Done starting upload Dec 16 17:22:28.355920163 (265h53m28.227003208s) [upload=233070]: Uploading segments... Dec 16 17:22:28.356090597 (265h53m28.227173632s) [upload=233070]: Got next segment. Inline: false Dec 16 17:22:28.356131766 (265h53m28.227214801s) [upload=233070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:22:28.356167273 (265h53m28.227250318s) [upload=233070]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:22:28.602538979 (265h53m28.473622024s) [seg_pos={0 0}, upload=233070]: Begin upload segment... Dec 16 17:22:28.602655190 (265h53m28.473738235s) [seg_pos={0 0}, upload=233070]: Done begin upload segment. Dec 16 17:22:28.602702590 (265h53m28.473785635s) [noise=true , address=77.90.31.11:4554 , piece_id=IMT5XXWJZL3LHQMWVMCIU7WPX2OGOFWQ6EH7HPOO3O4ANN6PVSIA, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.602802460 (265h53m28.473885505s) [noise=true , address=45.91.116.10:4081 , piece_id=F6X4FPZ3PYRL2V6B5BTZLL36JSDIFA2AZUMB2Q3TEBVHDOPDGBRA, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.603472637 (265h53m28.474555672s) [noise=true , address=45.91.116.13:4154 , piece_id=EWWYIQY4FLTYPK4GDOAQPT4CTSM4WZBIJB2TZILD65G5RD6LN4XA, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.603827322 (265h53m28.474910367s) [noise=true , address=45.91.116.7:4009 , piece_id=54XGXE5QJ6UZJBV67K4DBDMPXG6A72PPSE3UQ3OXHU53RFP6KZ3A, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.603896484 (265h53m28.474979519s) [noise=true , address=145.79.227.35:4481 , piece_id=R5KOPMCY42A45GH2HEAIHRUXQORFVSJEYW7YAYU23HEDH6OUBQWQ, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.604214349 (265h53m28.475297394s) [noise=true , address=145.79.227.46:4601 , piece_id=WV6ECF6MO4W65RNLWVHGCM4RC4I5E627UOTRQR6PTQ74ZJRIKD2A, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.604287829 (265h53m28.475370864s) [noise=true , address=45.91.116.11:4118 , piece_id=MXVHHSHA7CHNU7BQZVFE52JRATWAUBKFMAW662FQPPDIKANVGK2Q, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.604632165 (265h53m28.4757152s) [noise=true , address=45.91.116.14:4191 , piece_id=ELCLNIWDWE2STLW5U6FZL6OMIEMRGUSMEB4E3CTUX5V64RMXP2QQ, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.604661400 (265h53m28.475744446s) [noise=true , address=145.79.227.34:4907 , piece_id=C5GHGT5OGA7XRJIHJANVBAIF7Z6VMCDKI6WPEFTBDZPUKMIPLO5A, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.605019282 (265h53m28.476102317s) [noise=true , address=77.90.31.21:4877 , piece_id=MWYGIKFVXISWBUXVB4WEWBE3BOP4VCIOD6I3MBWBYCZEV7YCIHNQ, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.605098462 (265h53m28.476181507s) [noise=true , address=145.79.227.27:4691 , piece_id=DT5BT7KS4FTRIGYLM3OQWHULUYOZUNEP6FZ7XUBB2RMORPPUM5ZQ, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.605462636 (265h53m28.476545681s) [noise=true , address=145.79.227.52:4446 , piece_id=4RN2ZBGTULZ6VQSO4H6VEKIDEG2DLMDUW74LMUXOZGPXNBWF7GHQ, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.605494667 (265h53m28.476577712s) [noise=true , address=145.79.227.11:4732 , piece_id=3OSGIDEGOSVIMIBBTTRBMXJHGT626AC4GYDC3U2OY62SWB6FEV6A, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.605842790 (265h53m28.476925825s) [noise=true , address=45.91.116.8:4047 , piece_id=SOIENQXK3MCGBRBA6FOKCHEDEWIUMHHRUXO4FSPG2525XY4DXZCA, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.606015409 (265h53m28.477098454s) [noise=true , address=77.90.31.18:4808 , piece_id=QJT3MASV2WG5WBUSDLTETUHEL74QP4NWC5JFXVQQF5WYXY6DCOJA, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.606383579 (265h53m28.477466624s) [noise=true , address=45.91.116.5:4297 , piece_id=YT3FOYJEKBYTGDMA4Q2MKMJBLQOHQVPFZXSI322GZ7HGQDZ33ESA, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:29.753134117 (265h53m29.624217162s) [noise=true , address=77.90.31.21:4877 , piece_id=MWYGIKFVXISWBUXVB4WEWBE3BOP4VCIOD6I3MBWBYCZEV7YCIHNQ, seg_pos={0 0}, upload=233070]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 16 17:22:30.332323516 (265h53m30.203406561s) [upload=233070]: Next returned nil segment Dec 16 17:22:30.332340469 (265h53m30.203423514s) [upload=233070]: Waiting for error group managing segments... Dec 16 17:22:30.616216770 (265h53m30.487299815s) [noise=true , address=145.79.227.35:4481 , piece_id=R5KOPMCY42A45GH2HEAIHRUXQORFVSJEYW7YAYU23HEDH6OUBQWQ, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.622252243 (265h53m30.493335278s) [noise=true , address=145.79.227.52:4446 , piece_id=4RN2ZBGTULZ6VQSO4H6VEKIDEG2DLMDUW74LMUXOZGPXNBWF7GHQ, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.623047456 (265h53m30.494130491s) [noise=true , address=145.79.227.46:4601 , piece_id=WV6ECF6MO4W65RNLWVHGCM4RC4I5E627UOTRQR6PTQ74ZJRIKD2A, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.633939096 (265h53m30.505022131s) [noise=true , address=145.79.227.27:4691 , piece_id=DT5BT7KS4FTRIGYLM3OQWHULUYOZUNEP6FZ7XUBB2RMORPPUM5ZQ, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.676237018 (265h53m30.547320063s) [noise=true , address=145.79.227.34:4907 , piece_id=C5GHGT5OGA7XRJIHJANVBAIF7Z6VMCDKI6WPEFTBDZPUKMIPLO5A, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.684076116 (265h53m30.555159151s) [noise=true , address=145.79.227.11:4732 , piece_id=3OSGIDEGOSVIMIBBTTRBMXJHGT626AC4GYDC3U2OY62SWB6FEV6A, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.851675636 (265h53m30.722758681s) [noise=true , address=77.90.31.11:4554 , piece_id=IMT5XXWJZL3LHQMWVMCIU7WPX2OGOFWQ6EH7HPOO3O4ANN6PVSIA, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.923476907 (265h53m30.794559942s) [noise=true , address=45.91.116.11:4118 , piece_id=MXVHHSHA7CHNU7BQZVFE52JRATWAUBKFMAW662FQPPDIKANVGK2Q, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.927061062 (265h53m30.798144107s) [noise=true , address=45.91.116.8:4047 , piece_id=SOIENQXK3MCGBRBA6FOKCHEDEWIUMHHRUXO4FSPG2525XY4DXZCA, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.940531031 (265h53m30.811614067s) [noise=true , address=45.91.116.14:4191 , piece_id=ELCLNIWDWE2STLW5U6FZL6OMIEMRGUSMEB4E3CTUX5V64RMXP2QQ, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.956255193 (265h53m30.827338238s) [noise=true , address=45.91.116.5:4297 , piece_id=YT3FOYJEKBYTGDMA4Q2MKMJBLQOHQVPFZXSI322GZ7HGQDZ33ESA, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:31.004934157 (265h53m30.876017202s) [noise=true , address=45.91.116.7:4009 , piece_id=54XGXE5QJ6UZJBV67K4DBDMPXG6A72PPSE3UQ3OXHU53RFP6KZ3A, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:31.126466316 (265h53m30.997549361s) [noise=true , address=45.91.116.13:4154 , piece_id=EWWYIQY4FLTYPK4GDOAQPT4CTSM4WZBIJB2TZILD65G5RD6LN4XA, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:31.126490482 (265h53m30.997573517s) [seg_pos={0 0}, upload=233070]: Segment reached optimal threshold of 13 pieces. Dec 16 17:22:31.126883510 (265h53m30.997966556s) [noise=true , address=45.91.116.10:4081 , piece_id=F6X4FPZ3PYRL2V6B5BTZLL36JSDIFA2AZUMB2Q3TEBVHDOPDGBRA, seg_pos={0 0}, upload=233070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:60060->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:60060->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60060->45.91.116.10:4081: use of closed network connection Dec 16 17:22:31.126900472 (265h53m30.997983517s) [noise=true , address=45.91.116.10:4081 , piece_id=F6X4FPZ3PYRL2V6B5BTZLL36JSDIFA2AZUMB2Q3TEBVHDOPDGBRA, seg_pos={0 0}, upload=233070]: Long tail cancellation due to optimal threshold Dec 16 17:22:31.126980125 (265h53m30.99806317s) [noise=true , address=77.90.31.18:4808 , piece_id=QJT3MASV2WG5WBUSDLTETUHEL74QP4NWC5JFXVQQF5WYXY6DCOJA, seg_pos={0 0}, upload=233070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:35310->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:35310->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35310->77.90.31.18:4808: use of closed network connection Dec 16 17:22:31.126991246 (265h53m30.998074291s) [noise=true , address=77.90.31.18:4808 , piece_id=QJT3MASV2WG5WBUSDLTETUHEL74QP4NWC5JFXVQQF5WYXY6DCOJA, seg_pos={0 0}, upload=233070]: Long tail cancellation due to optimal threshold Dec 16 17:22:31.127010171 (265h53m30.998093216s) [seg_pos={0 0}, upload=233070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:22:31.127091877 (265h53m30.998174912s) [upload=233070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:22:31.127098369 (265h53m30.998181404s) [upload=233070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:22:31.423343183 (265h53m31.294426218s) [upload=233070]: Done uploading segments... Dec 16 17:22:31.423362178 (265h53m31.294445213s) [upload=233070]: Upload finished. err: <nil> Duration: 6.025762424s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [...X..X....X................XX....X...................XX....] duration: 4.896536627s trace: https://jaeger.dev.storj.io/trace/2d7e37a5218d47ae FAIL (2025-12-16 17:26:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.896628373s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X...................................................X....] duration: 24.524296987s trace: https://jaeger.dev.storj.io/trace/300ebb63ed4f1615 FAIL (2025-12-16 17:26:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 16 17:22:28.355769386 (265h53m28.226852431s) [upload=233070]: Starting upload Dec 16 17:22:28.355830982 (265h53m28.226914028s) [upload=233070]: Done starting upload Dec 16 17:22:28.355920163 (265h53m28.227003208s) [upload=233070]: Uploading segments... Dec 16 17:22:28.356090597 (265h53m28.227173632s) [upload=233070]: Got next segment. Inline: false Dec 16 17:22:28.356131766 (265h53m28.227214801s) [upload=233070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:22:28.356167273 (265h53m28.227250318s) [upload=233070]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:22:28.602538979 (265h53m28.473622024s) [seg_pos={0 0}, upload=233070]: Begin upload segment... Dec 16 17:22:28.602655190 (265h53m28.473738235s) [seg_pos={0 0}, upload=233070]: Done begin upload segment. Dec 16 17:22:28.602702590 (265h53m28.473785635s) [noise=true , address=77.90.31.11:4554 , piece_id=IMT5XXWJZL3LHQMWVMCIU7WPX2OGOFWQ6EH7HPOO3O4ANN6PVSIA, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.602802460 (265h53m28.473885505s) [noise=true , address=45.91.116.10:4081 , piece_id=F6X4FPZ3PYRL2V6B5BTZLL36JSDIFA2AZUMB2Q3TEBVHDOPDGBRA, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.603472637 (265h53m28.474555672s) [noise=true , address=45.91.116.13:4154 , piece_id=EWWYIQY4FLTYPK4GDOAQPT4CTSM4WZBIJB2TZILD65G5RD6LN4XA, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.603827322 (265h53m28.474910367s) [noise=true , address=45.91.116.7:4009 , piece_id=54XGXE5QJ6UZJBV67K4DBDMPXG6A72PPSE3UQ3OXHU53RFP6KZ3A, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.603896484 (265h53m28.474979519s) [noise=true , address=145.79.227.35:4481 , piece_id=R5KOPMCY42A45GH2HEAIHRUXQORFVSJEYW7YAYU23HEDH6OUBQWQ, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.604214349 (265h53m28.475297394s) [noise=true , address=145.79.227.46:4601 , piece_id=WV6ECF6MO4W65RNLWVHGCM4RC4I5E627UOTRQR6PTQ74ZJRIKD2A, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.604287829 (265h53m28.475370864s) [noise=true , address=45.91.116.11:4118 , piece_id=MXVHHSHA7CHNU7BQZVFE52JRATWAUBKFMAW662FQPPDIKANVGK2Q, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.604632165 (265h53m28.4757152s) [noise=true , address=45.91.116.14:4191 , piece_id=ELCLNIWDWE2STLW5U6FZL6OMIEMRGUSMEB4E3CTUX5V64RMXP2QQ, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.604661400 (265h53m28.475744446s) [noise=true , address=145.79.227.34:4907 , piece_id=C5GHGT5OGA7XRJIHJANVBAIF7Z6VMCDKI6WPEFTBDZPUKMIPLO5A, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.605019282 (265h53m28.476102317s) [noise=true , address=77.90.31.21:4877 , piece_id=MWYGIKFVXISWBUXVB4WEWBE3BOP4VCIOD6I3MBWBYCZEV7YCIHNQ, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.605098462 (265h53m28.476181507s) [noise=true , address=145.79.227.27:4691 , piece_id=DT5BT7KS4FTRIGYLM3OQWHULUYOZUNEP6FZ7XUBB2RMORPPUM5ZQ, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.605462636 (265h53m28.476545681s) [noise=true , address=145.79.227.52:4446 , piece_id=4RN2ZBGTULZ6VQSO4H6VEKIDEG2DLMDUW74LMUXOZGPXNBWF7GHQ, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.605494667 (265h53m28.476577712s) [noise=true , address=145.79.227.11:4732 , piece_id=3OSGIDEGOSVIMIBBTTRBMXJHGT626AC4GYDC3U2OY62SWB6FEV6A, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.605842790 (265h53m28.476925825s) [noise=true , address=45.91.116.8:4047 , piece_id=SOIENQXK3MCGBRBA6FOKCHEDEWIUMHHRUXO4FSPG2525XY4DXZCA, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.606015409 (265h53m28.477098454s) [noise=true , address=77.90.31.18:4808 , piece_id=QJT3MASV2WG5WBUSDLTETUHEL74QP4NWC5JFXVQQF5WYXY6DCOJA, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.606383579 (265h53m28.477466624s) [noise=true , address=45.91.116.5:4297 , piece_id=YT3FOYJEKBYTGDMA4Q2MKMJBLQOHQVPFZXSI322GZ7HGQDZ33ESA, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:29.753134117 (265h53m29.624217162s) [noise=true , address=77.90.31.21:4877 , piece_id=MWYGIKFVXISWBUXVB4WEWBE3BOP4VCIOD6I3MBWBYCZEV7YCIHNQ, seg_pos={0 0}, upload=233070]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 16 17:22:30.332323516 (265h53m30.203406561s) [upload=233070]: Next returned nil segment Dec 16 17:22:30.332340469 (265h53m30.203423514s) [upload=233070]: Waiting for error group managing segments... Dec 16 17:22:30.616216770 (265h53m30.487299815s) [noise=true , address=145.79.227.35:4481 , piece_id=R5KOPMCY42A45GH2HEAIHRUXQORFVSJEYW7YAYU23HEDH6OUBQWQ, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.622252243 (265h53m30.493335278s) [noise=true , address=145.79.227.52:4446 , piece_id=4RN2ZBGTULZ6VQSO4H6VEKIDEG2DLMDUW74LMUXOZGPXNBWF7GHQ, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.623047456 (265h53m30.494130491s) [noise=true , address=145.79.227.46:4601 , piece_id=WV6ECF6MO4W65RNLWVHGCM4RC4I5E627UOTRQR6PTQ74ZJRIKD2A, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.633939096 (265h53m30.505022131s) [noise=true , address=145.79.227.27:4691 , piece_id=DT5BT7KS4FTRIGYLM3OQWHULUYOZUNEP6FZ7XUBB2RMORPPUM5ZQ, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.676237018 (265h53m30.547320063s) [noise=true , address=145.79.227.34:4907 , piece_id=C5GHGT5OGA7XRJIHJANVBAIF7Z6VMCDKI6WPEFTBDZPUKMIPLO5A, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.684076116 (265h53m30.555159151s) [noise=true , address=145.79.227.11:4732 , piece_id=3OSGIDEGOSVIMIBBTTRBMXJHGT626AC4GYDC3U2OY62SWB6FEV6A, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.851675636 (265h53m30.722758681s) [noise=true , address=77.90.31.11:4554 , piece_id=IMT5XXWJZL3LHQMWVMCIU7WPX2OGOFWQ6EH7HPOO3O4ANN6PVSIA, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.923476907 (265h53m30.794559942s) [noise=true , address=45.91.116.11:4118 , piece_id=MXVHHSHA7CHNU7BQZVFE52JRATWAUBKFMAW662FQPPDIKANVGK2Q, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.927061062 (265h53m30.798144107s) [noise=true , address=45.91.116.8:4047 , piece_id=SOIENQXK3MCGBRBA6FOKCHEDEWIUMHHRUXO4FSPG2525XY4DXZCA, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.940531031 (265h53m30.811614067s) [noise=true , address=45.91.116.14:4191 , piece_id=ELCLNIWDWE2STLW5U6FZL6OMIEMRGUSMEB4E3CTUX5V64RMXP2QQ, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.956255193 (265h53m30.827338238s) [noise=true , address=45.91.116.5:4297 , piece_id=YT3FOYJEKBYTGDMA4Q2MKMJBLQOHQVPFZXSI322GZ7HGQDZ33ESA, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:31.004934157 (265h53m30.876017202s) [noise=true , address=45.91.116.7:4009 , piece_id=54XGXE5QJ6UZJBV67K4DBDMPXG6A72PPSE3UQ3OXHU53RFP6KZ3A, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:31.126466316 (265h53m30.997549361s) [noise=true , address=45.91.116.13:4154 , piece_id=EWWYIQY4FLTYPK4GDOAQPT4CTSM4WZBIJB2TZILD65G5RD6LN4XA, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:31.126490482 (265h53m30.997573517s) [seg_pos={0 0}, upload=233070]: Segment reached optimal threshold of 13 pieces. Dec 16 17:22:31.126883510 (265h53m30.997966556s) [noise=true , address=45.91.116.10:4081 , piece_id=F6X4FPZ3PYRL2V6B5BTZLL36JSDIFA2AZUMB2Q3TEBVHDOPDGBRA, seg_pos={0 0}, upload=233070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:60060->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:60060->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60060->45.91.116.10:4081: use of closed network connection Dec 16 17:22:31.126900472 (265h53m30.997983517s) [noise=true , address=45.91.116.10:4081 , piece_id=F6X4FPZ3PYRL2V6B5BTZLL36JSDIFA2AZUMB2Q3TEBVHDOPDGBRA, seg_pos={0 0}, upload=233070]: Long tail cancellation due to optimal threshold Dec 16 17:22:31.126980125 (265h53m30.99806317s) [noise=true , address=77.90.31.18:4808 , piece_id=QJT3MASV2WG5WBUSDLTETUHEL74QP4NWC5JFXVQQF5WYXY6DCOJA, seg_pos={0 0}, upload=233070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:35310->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:35310->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35310->77.90.31.18:4808: use of closed network connection Dec 16 17:22:31.126991246 (265h53m30.998074291s) [noise=true , address=77.90.31.18:4808 , piece_id=QJT3MASV2WG5WBUSDLTETUHEL74QP4NWC5JFXVQQF5WYXY6DCOJA, seg_pos={0 0}, upload=233070]: Long tail cancellation due to optimal threshold Dec 16 17:22:31.127010171 (265h53m30.998093216s) [seg_pos={0 0}, upload=233070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:22:31.127091877 (265h53m30.998174912s) [upload=233070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:22:31.127098369 (265h53m30.998181404s) [upload=233070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:22:31.423343183 (265h53m31.294426218s) [upload=233070]: Done uploading segments... Dec 16 17:22:31.423362178 (265h53m31.294445213s) [upload=233070]: Upload finished. err: <nil> Duration: 24.524338947s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [....X..X....X................XX....X...................XX...] duration: 5.000591775s trace: https://jaeger.dev.storj.io/trace/2d7e37a5218d47ae FAIL (2025-12-16 17:25:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000616642s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.395670664s trace: https://jaeger.dev.storj.io/trace/56c5b90ed930ab07 FAIL (2025-12-16 17:25:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.395696984s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X...................................................X...] duration: 5.897054269s trace: https://jaeger.dev.storj.io/trace/300ebb63ed4f1615 FAIL (2025-12-16 17:25:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 16 17:22:28.355769386 (265h53m28.226852431s) [upload=233070]: Starting upload Dec 16 17:22:28.355830982 (265h53m28.226914028s) [upload=233070]: Done starting upload Dec 16 17:22:28.355920163 (265h53m28.227003208s) [upload=233070]: Uploading segments... Dec 16 17:22:28.356090597 (265h53m28.227173632s) [upload=233070]: Got next segment. Inline: false Dec 16 17:22:28.356131766 (265h53m28.227214801s) [upload=233070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:22:28.356167273 (265h53m28.227250318s) [upload=233070]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:22:28.602538979 (265h53m28.473622024s) [seg_pos={0 0}, upload=233070]: Begin upload segment... Dec 16 17:22:28.602655190 (265h53m28.473738235s) [seg_pos={0 0}, upload=233070]: Done begin upload segment. Dec 16 17:22:28.602702590 (265h53m28.473785635s) [noise=true , address=77.90.31.11:4554 , piece_id=IMT5XXWJZL3LHQMWVMCIU7WPX2OGOFWQ6EH7HPOO3O4ANN6PVSIA, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.602802460 (265h53m28.473885505s) [noise=true , address=45.91.116.10:4081 , piece_id=F6X4FPZ3PYRL2V6B5BTZLL36JSDIFA2AZUMB2Q3TEBVHDOPDGBRA, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.603472637 (265h53m28.474555672s) [noise=true , address=45.91.116.13:4154 , piece_id=EWWYIQY4FLTYPK4GDOAQPT4CTSM4WZBIJB2TZILD65G5RD6LN4XA, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.603827322 (265h53m28.474910367s) [noise=true , address=45.91.116.7:4009 , piece_id=54XGXE5QJ6UZJBV67K4DBDMPXG6A72PPSE3UQ3OXHU53RFP6KZ3A, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.603896484 (265h53m28.474979519s) [noise=true , address=145.79.227.35:4481 , piece_id=R5KOPMCY42A45GH2HEAIHRUXQORFVSJEYW7YAYU23HEDH6OUBQWQ, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.604214349 (265h53m28.475297394s) [noise=true , address=145.79.227.46:4601 , piece_id=WV6ECF6MO4W65RNLWVHGCM4RC4I5E627UOTRQR6PTQ74ZJRIKD2A, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.604287829 (265h53m28.475370864s) [noise=true , address=45.91.116.11:4118 , piece_id=MXVHHSHA7CHNU7BQZVFE52JRATWAUBKFMAW662FQPPDIKANVGK2Q, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.604632165 (265h53m28.4757152s) [noise=true , address=45.91.116.14:4191 , piece_id=ELCLNIWDWE2STLW5U6FZL6OMIEMRGUSMEB4E3CTUX5V64RMXP2QQ, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.604661400 (265h53m28.475744446s) [noise=true , address=145.79.227.34:4907 , piece_id=C5GHGT5OGA7XRJIHJANVBAIF7Z6VMCDKI6WPEFTBDZPUKMIPLO5A, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.605019282 (265h53m28.476102317s) [noise=true , address=77.90.31.21:4877 , piece_id=MWYGIKFVXISWBUXVB4WEWBE3BOP4VCIOD6I3MBWBYCZEV7YCIHNQ, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.605098462 (265h53m28.476181507s) [noise=true , address=145.79.227.27:4691 , piece_id=DT5BT7KS4FTRIGYLM3OQWHULUYOZUNEP6FZ7XUBB2RMORPPUM5ZQ, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.605462636 (265h53m28.476545681s) [noise=true , address=145.79.227.52:4446 , piece_id=4RN2ZBGTULZ6VQSO4H6VEKIDEG2DLMDUW74LMUXOZGPXNBWF7GHQ, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.605494667 (265h53m28.476577712s) [noise=true , address=145.79.227.11:4732 , piece_id=3OSGIDEGOSVIMIBBTTRBMXJHGT626AC4GYDC3U2OY62SWB6FEV6A, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.605842790 (265h53m28.476925825s) [noise=true , address=45.91.116.8:4047 , piece_id=SOIENQXK3MCGBRBA6FOKCHEDEWIUMHHRUXO4FSPG2525XY4DXZCA, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.606015409 (265h53m28.477098454s) [noise=true , address=77.90.31.18:4808 , piece_id=QJT3MASV2WG5WBUSDLTETUHEL74QP4NWC5JFXVQQF5WYXY6DCOJA, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.606383579 (265h53m28.477466624s) [noise=true , address=45.91.116.5:4297 , piece_id=YT3FOYJEKBYTGDMA4Q2MKMJBLQOHQVPFZXSI322GZ7HGQDZ33ESA, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:29.753134117 (265h53m29.624217162s) [noise=true , address=77.90.31.21:4877 , piece_id=MWYGIKFVXISWBUXVB4WEWBE3BOP4VCIOD6I3MBWBYCZEV7YCIHNQ, seg_pos={0 0}, upload=233070]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 16 17:22:30.332323516 (265h53m30.203406561s) [upload=233070]: Next returned nil segment Dec 16 17:22:30.332340469 (265h53m30.203423514s) [upload=233070]: Waiting for error group managing segments... Dec 16 17:22:30.616216770 (265h53m30.487299815s) [noise=true , address=145.79.227.35:4481 , piece_id=R5KOPMCY42A45GH2HEAIHRUXQORFVSJEYW7YAYU23HEDH6OUBQWQ, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.622252243 (265h53m30.493335278s) [noise=true , address=145.79.227.52:4446 , piece_id=4RN2ZBGTULZ6VQSO4H6VEKIDEG2DLMDUW74LMUXOZGPXNBWF7GHQ, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.623047456 (265h53m30.494130491s) [noise=true , address=145.79.227.46:4601 , piece_id=WV6ECF6MO4W65RNLWVHGCM4RC4I5E627UOTRQR6PTQ74ZJRIKD2A, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.633939096 (265h53m30.505022131s) [noise=true , address=145.79.227.27:4691 , piece_id=DT5BT7KS4FTRIGYLM3OQWHULUYOZUNEP6FZ7XUBB2RMORPPUM5ZQ, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.676237018 (265h53m30.547320063s) [noise=true , address=145.79.227.34:4907 , piece_id=C5GHGT5OGA7XRJIHJANVBAIF7Z6VMCDKI6WPEFTBDZPUKMIPLO5A, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.684076116 (265h53m30.555159151s) [noise=true , address=145.79.227.11:4732 , piece_id=3OSGIDEGOSVIMIBBTTRBMXJHGT626AC4GYDC3U2OY62SWB6FEV6A, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.851675636 (265h53m30.722758681s) [noise=true , address=77.90.31.11:4554 , piece_id=IMT5XXWJZL3LHQMWVMCIU7WPX2OGOFWQ6EH7HPOO3O4ANN6PVSIA, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.923476907 (265h53m30.794559942s) [noise=true , address=45.91.116.11:4118 , piece_id=MXVHHSHA7CHNU7BQZVFE52JRATWAUBKFMAW662FQPPDIKANVGK2Q, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.927061062 (265h53m30.798144107s) [noise=true , address=45.91.116.8:4047 , piece_id=SOIENQXK3MCGBRBA6FOKCHEDEWIUMHHRUXO4FSPG2525XY4DXZCA, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.940531031 (265h53m30.811614067s) [noise=true , address=45.91.116.14:4191 , piece_id=ELCLNIWDWE2STLW5U6FZL6OMIEMRGUSMEB4E3CTUX5V64RMXP2QQ, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.956255193 (265h53m30.827338238s) [noise=true , address=45.91.116.5:4297 , piece_id=YT3FOYJEKBYTGDMA4Q2MKMJBLQOHQVPFZXSI322GZ7HGQDZ33ESA, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:31.004934157 (265h53m30.876017202s) [noise=true , address=45.91.116.7:4009 , piece_id=54XGXE5QJ6UZJBV67K4DBDMPXG6A72PPSE3UQ3OXHU53RFP6KZ3A, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:31.126466316 (265h53m30.997549361s) [noise=true , address=45.91.116.13:4154 , piece_id=EWWYIQY4FLTYPK4GDOAQPT4CTSM4WZBIJB2TZILD65G5RD6LN4XA, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:31.126490482 (265h53m30.997573517s) [seg_pos={0 0}, upload=233070]: Segment reached optimal threshold of 13 pieces. Dec 16 17:22:31.126883510 (265h53m30.997966556s) [noise=true , address=45.91.116.10:4081 , piece_id=F6X4FPZ3PYRL2V6B5BTZLL36JSDIFA2AZUMB2Q3TEBVHDOPDGBRA, seg_pos={0 0}, upload=233070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:60060->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:60060->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60060->45.91.116.10:4081: use of closed network connection Dec 16 17:22:31.126900472 (265h53m30.997983517s) [noise=true , address=45.91.116.10:4081 , piece_id=F6X4FPZ3PYRL2V6B5BTZLL36JSDIFA2AZUMB2Q3TEBVHDOPDGBRA, seg_pos={0 0}, upload=233070]: Long tail cancellation due to optimal threshold Dec 16 17:22:31.126980125 (265h53m30.99806317s) [noise=true , address=77.90.31.18:4808 , piece_id=QJT3MASV2WG5WBUSDLTETUHEL74QP4NWC5JFXVQQF5WYXY6DCOJA, seg_pos={0 0}, upload=233070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:35310->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:35310->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35310->77.90.31.18:4808: use of closed network connection Dec 16 17:22:31.126991246 (265h53m30.998074291s) [noise=true , address=77.90.31.18:4808 , piece_id=QJT3MASV2WG5WBUSDLTETUHEL74QP4NWC5JFXVQQF5WYXY6DCOJA, seg_pos={0 0}, upload=233070]: Long tail cancellation due to optimal threshold Dec 16 17:22:31.127010171 (265h53m30.998093216s) [seg_pos={0 0}, upload=233070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:22:31.127091877 (265h53m30.998174912s) [upload=233070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:22:31.127098369 (265h53m30.998181404s) [upload=233070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:22:31.423343183 (265h53m31.294426218s) [upload=233070]: Done uploading segments... Dec 16 17:22:31.423362178 (265h53m31.294445213s) [upload=233070]: Upload finished. err: <nil> Duration: 5.897074448s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.621921757s trace: https://jaeger.dev.storj.io/trace/56c5b90ed930ab07 FAIL (2025-12-16 17:25:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.62195597s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.....X..X....X................XX....X...................XX..] duration: 3.448504687s trace: https://jaeger.dev.storj.io/trace/2d7e37a5218d47ae FAIL (2025-12-16 17:24:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.448528782s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X...................................................X..] duration: 6.192802808s trace: https://jaeger.dev.storj.io/trace/300ebb63ed4f1615 FAIL (2025-12-16 17:24:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 16 17:22:28.355769386 (265h53m28.226852431s) [upload=233070]: Starting upload Dec 16 17:22:28.355830982 (265h53m28.226914028s) [upload=233070]: Done starting upload Dec 16 17:22:28.355920163 (265h53m28.227003208s) [upload=233070]: Uploading segments... Dec 16 17:22:28.356090597 (265h53m28.227173632s) [upload=233070]: Got next segment. Inline: false Dec 16 17:22:28.356131766 (265h53m28.227214801s) [upload=233070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:22:28.356167273 (265h53m28.227250318s) [upload=233070]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:22:28.602538979 (265h53m28.473622024s) [seg_pos={0 0}, upload=233070]: Begin upload segment... Dec 16 17:22:28.602655190 (265h53m28.473738235s) [seg_pos={0 0}, upload=233070]: Done begin upload segment. Dec 16 17:22:28.602702590 (265h53m28.473785635s) [noise=true , address=77.90.31.11:4554 , piece_id=IMT5XXWJZL3LHQMWVMCIU7WPX2OGOFWQ6EH7HPOO3O4ANN6PVSIA, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.602802460 (265h53m28.473885505s) [noise=true , address=45.91.116.10:4081 , piece_id=F6X4FPZ3PYRL2V6B5BTZLL36JSDIFA2AZUMB2Q3TEBVHDOPDGBRA, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.603472637 (265h53m28.474555672s) [noise=true , address=45.91.116.13:4154 , piece_id=EWWYIQY4FLTYPK4GDOAQPT4CTSM4WZBIJB2TZILD65G5RD6LN4XA, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.603827322 (265h53m28.474910367s) [noise=true , address=45.91.116.7:4009 , piece_id=54XGXE5QJ6UZJBV67K4DBDMPXG6A72PPSE3UQ3OXHU53RFP6KZ3A, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.603896484 (265h53m28.474979519s) [noise=true , address=145.79.227.35:4481 , piece_id=R5KOPMCY42A45GH2HEAIHRUXQORFVSJEYW7YAYU23HEDH6OUBQWQ, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.604214349 (265h53m28.475297394s) [noise=true , address=145.79.227.46:4601 , piece_id=WV6ECF6MO4W65RNLWVHGCM4RC4I5E627UOTRQR6PTQ74ZJRIKD2A, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.604287829 (265h53m28.475370864s) [noise=true , address=45.91.116.11:4118 , piece_id=MXVHHSHA7CHNU7BQZVFE52JRATWAUBKFMAW662FQPPDIKANVGK2Q, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.604632165 (265h53m28.4757152s) [noise=true , address=45.91.116.14:4191 , piece_id=ELCLNIWDWE2STLW5U6FZL6OMIEMRGUSMEB4E3CTUX5V64RMXP2QQ, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.604661400 (265h53m28.475744446s) [noise=true , address=145.79.227.34:4907 , piece_id=C5GHGT5OGA7XRJIHJANVBAIF7Z6VMCDKI6WPEFTBDZPUKMIPLO5A, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.605019282 (265h53m28.476102317s) [noise=true , address=77.90.31.21:4877 , piece_id=MWYGIKFVXISWBUXVB4WEWBE3BOP4VCIOD6I3MBWBYCZEV7YCIHNQ, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.605098462 (265h53m28.476181507s) [noise=true , address=145.79.227.27:4691 , piece_id=DT5BT7KS4FTRIGYLM3OQWHULUYOZUNEP6FZ7XUBB2RMORPPUM5ZQ, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.605462636 (265h53m28.476545681s) [noise=true , address=145.79.227.52:4446 , piece_id=4RN2ZBGTULZ6VQSO4H6VEKIDEG2DLMDUW74LMUXOZGPXNBWF7GHQ, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.605494667 (265h53m28.476577712s) [noise=true , address=145.79.227.11:4732 , piece_id=3OSGIDEGOSVIMIBBTTRBMXJHGT626AC4GYDC3U2OY62SWB6FEV6A, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.605842790 (265h53m28.476925825s) [noise=true , address=45.91.116.8:4047 , piece_id=SOIENQXK3MCGBRBA6FOKCHEDEWIUMHHRUXO4FSPG2525XY4DXZCA, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.606015409 (265h53m28.477098454s) [noise=true , address=77.90.31.18:4808 , piece_id=QJT3MASV2WG5WBUSDLTETUHEL74QP4NWC5JFXVQQF5WYXY6DCOJA, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.606383579 (265h53m28.477466624s) [noise=true , address=45.91.116.5:4297 , piece_id=YT3FOYJEKBYTGDMA4Q2MKMJBLQOHQVPFZXSI322GZ7HGQDZ33ESA, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:29.753134117 (265h53m29.624217162s) [noise=true , address=77.90.31.21:4877 , piece_id=MWYGIKFVXISWBUXVB4WEWBE3BOP4VCIOD6I3MBWBYCZEV7YCIHNQ, seg_pos={0 0}, upload=233070]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 16 17:22:30.332323516 (265h53m30.203406561s) [upload=233070]: Next returned nil segment Dec 16 17:22:30.332340469 (265h53m30.203423514s) [upload=233070]: Waiting for error group managing segments... Dec 16 17:22:30.616216770 (265h53m30.487299815s) [noise=true , address=145.79.227.35:4481 , piece_id=R5KOPMCY42A45GH2HEAIHRUXQORFVSJEYW7YAYU23HEDH6OUBQWQ, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.622252243 (265h53m30.493335278s) [noise=true , address=145.79.227.52:4446 , piece_id=4RN2ZBGTULZ6VQSO4H6VEKIDEG2DLMDUW74LMUXOZGPXNBWF7GHQ, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.623047456 (265h53m30.494130491s) [noise=true , address=145.79.227.46:4601 , piece_id=WV6ECF6MO4W65RNLWVHGCM4RC4I5E627UOTRQR6PTQ74ZJRIKD2A, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.633939096 (265h53m30.505022131s) [noise=true , address=145.79.227.27:4691 , piece_id=DT5BT7KS4FTRIGYLM3OQWHULUYOZUNEP6FZ7XUBB2RMORPPUM5ZQ, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.676237018 (265h53m30.547320063s) [noise=true , address=145.79.227.34:4907 , piece_id=C5GHGT5OGA7XRJIHJANVBAIF7Z6VMCDKI6WPEFTBDZPUKMIPLO5A, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.684076116 (265h53m30.555159151s) [noise=true , address=145.79.227.11:4732 , piece_id=3OSGIDEGOSVIMIBBTTRBMXJHGT626AC4GYDC3U2OY62SWB6FEV6A, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.851675636 (265h53m30.722758681s) [noise=true , address=77.90.31.11:4554 , piece_id=IMT5XXWJZL3LHQMWVMCIU7WPX2OGOFWQ6EH7HPOO3O4ANN6PVSIA, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.923476907 (265h53m30.794559942s) [noise=true , address=45.91.116.11:4118 , piece_id=MXVHHSHA7CHNU7BQZVFE52JRATWAUBKFMAW662FQPPDIKANVGK2Q, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.927061062 (265h53m30.798144107s) [noise=true , address=45.91.116.8:4047 , piece_id=SOIENQXK3MCGBRBA6FOKCHEDEWIUMHHRUXO4FSPG2525XY4DXZCA, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.940531031 (265h53m30.811614067s) [noise=true , address=45.91.116.14:4191 , piece_id=ELCLNIWDWE2STLW5U6FZL6OMIEMRGUSMEB4E3CTUX5V64RMXP2QQ, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.956255193 (265h53m30.827338238s) [noise=true , address=45.91.116.5:4297 , piece_id=YT3FOYJEKBYTGDMA4Q2MKMJBLQOHQVPFZXSI322GZ7HGQDZ33ESA, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:31.004934157 (265h53m30.876017202s) [noise=true , address=45.91.116.7:4009 , piece_id=54XGXE5QJ6UZJBV67K4DBDMPXG6A72PPSE3UQ3OXHU53RFP6KZ3A, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:31.126466316 (265h53m30.997549361s) [noise=true , address=45.91.116.13:4154 , piece_id=EWWYIQY4FLTYPK4GDOAQPT4CTSM4WZBIJB2TZILD65G5RD6LN4XA, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:31.126490482 (265h53m30.997573517s) [seg_pos={0 0}, upload=233070]: Segment reached optimal threshold of 13 pieces. Dec 16 17:22:31.126883510 (265h53m30.997966556s) [noise=true , address=45.91.116.10:4081 , piece_id=F6X4FPZ3PYRL2V6B5BTZLL36JSDIFA2AZUMB2Q3TEBVHDOPDGBRA, seg_pos={0 0}, upload=233070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:60060->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:60060->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60060->45.91.116.10:4081: use of closed network connection Dec 16 17:22:31.126900472 (265h53m30.997983517s) [noise=true , address=45.91.116.10:4081 , piece_id=F6X4FPZ3PYRL2V6B5BTZLL36JSDIFA2AZUMB2Q3TEBVHDOPDGBRA, seg_pos={0 0}, upload=233070]: Long tail cancellation due to optimal threshold Dec 16 17:22:31.126980125 (265h53m30.99806317s) [noise=true , address=77.90.31.18:4808 , piece_id=QJT3MASV2WG5WBUSDLTETUHEL74QP4NWC5JFXVQQF5WYXY6DCOJA, seg_pos={0 0}, upload=233070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:35310->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:35310->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35310->77.90.31.18:4808: use of closed network connection Dec 16 17:22:31.126991246 (265h53m30.998074291s) [noise=true , address=77.90.31.18:4808 , piece_id=QJT3MASV2WG5WBUSDLTETUHEL74QP4NWC5JFXVQQF5WYXY6DCOJA, seg_pos={0 0}, upload=233070]: Long tail cancellation due to optimal threshold Dec 16 17:22:31.127010171 (265h53m30.998093216s) [seg_pos={0 0}, upload=233070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:22:31.127091877 (265h53m30.998174912s) [upload=233070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:22:31.127098369 (265h53m30.998181404s) [upload=233070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:22:31.423343183 (265h53m31.294426218s) [upload=233070]: Done uploading segments... Dec 16 17:22:31.423362178 (265h53m31.294445213s) [upload=233070]: Upload finished. err: <nil> Duration: 6.192831763s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.314621504s trace: https://jaeger.dev.storj.io/trace/56c5b90ed930ab07 FAIL (2025-12-16 17:24:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.314649348s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [......X..X....X................XX....X...................XX.] duration: 4.682695957s trace: https://jaeger.dev.storj.io/trace/2d7e37a5218d47ae FAIL (2025-12-16 17:23:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.682733328s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.262457572s trace: https://jaeger.dev.storj.io/trace/56c5b90ed930ab07 FAIL (2025-12-16 17:23:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.262488701s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X...................................................X.] duration: 5.803788575s trace: https://jaeger.dev.storj.io/trace/300ebb63ed4f1615 FAIL (2025-12-16 17:23:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 16 17:22:28.355769386 (265h53m28.226852431s) [upload=233070]: Starting upload Dec 16 17:22:28.355830982 (265h53m28.226914028s) [upload=233070]: Done starting upload Dec 16 17:22:28.355920163 (265h53m28.227003208s) [upload=233070]: Uploading segments... Dec 16 17:22:28.356090597 (265h53m28.227173632s) [upload=233070]: Got next segment. Inline: false Dec 16 17:22:28.356131766 (265h53m28.227214801s) [upload=233070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:22:28.356167273 (265h53m28.227250318s) [upload=233070]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:22:28.602538979 (265h53m28.473622024s) [seg_pos={0 0}, upload=233070]: Begin upload segment... Dec 16 17:22:28.602655190 (265h53m28.473738235s) [seg_pos={0 0}, upload=233070]: Done begin upload segment. Dec 16 17:22:28.602702590 (265h53m28.473785635s) [noise=true , address=77.90.31.11:4554 , piece_id=IMT5XXWJZL3LHQMWVMCIU7WPX2OGOFWQ6EH7HPOO3O4ANN6PVSIA, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.602802460 (265h53m28.473885505s) [noise=true , address=45.91.116.10:4081 , piece_id=F6X4FPZ3PYRL2V6B5BTZLL36JSDIFA2AZUMB2Q3TEBVHDOPDGBRA, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.603472637 (265h53m28.474555672s) [noise=true , address=45.91.116.13:4154 , piece_id=EWWYIQY4FLTYPK4GDOAQPT4CTSM4WZBIJB2TZILD65G5RD6LN4XA, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.603827322 (265h53m28.474910367s) [noise=true , address=45.91.116.7:4009 , piece_id=54XGXE5QJ6UZJBV67K4DBDMPXG6A72PPSE3UQ3OXHU53RFP6KZ3A, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.603896484 (265h53m28.474979519s) [noise=true , address=145.79.227.35:4481 , piece_id=R5KOPMCY42A45GH2HEAIHRUXQORFVSJEYW7YAYU23HEDH6OUBQWQ, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.604214349 (265h53m28.475297394s) [noise=true , address=145.79.227.46:4601 , piece_id=WV6ECF6MO4W65RNLWVHGCM4RC4I5E627UOTRQR6PTQ74ZJRIKD2A, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.604287829 (265h53m28.475370864s) [noise=true , address=45.91.116.11:4118 , piece_id=MXVHHSHA7CHNU7BQZVFE52JRATWAUBKFMAW662FQPPDIKANVGK2Q, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.604632165 (265h53m28.4757152s) [noise=true , address=45.91.116.14:4191 , piece_id=ELCLNIWDWE2STLW5U6FZL6OMIEMRGUSMEB4E3CTUX5V64RMXP2QQ, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.604661400 (265h53m28.475744446s) [noise=true , address=145.79.227.34:4907 , piece_id=C5GHGT5OGA7XRJIHJANVBAIF7Z6VMCDKI6WPEFTBDZPUKMIPLO5A, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.605019282 (265h53m28.476102317s) [noise=true , address=77.90.31.21:4877 , piece_id=MWYGIKFVXISWBUXVB4WEWBE3BOP4VCIOD6I3MBWBYCZEV7YCIHNQ, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.605098462 (265h53m28.476181507s) [noise=true , address=145.79.227.27:4691 , piece_id=DT5BT7KS4FTRIGYLM3OQWHULUYOZUNEP6FZ7XUBB2RMORPPUM5ZQ, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.605462636 (265h53m28.476545681s) [noise=true , address=145.79.227.52:4446 , piece_id=4RN2ZBGTULZ6VQSO4H6VEKIDEG2DLMDUW74LMUXOZGPXNBWF7GHQ, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.605494667 (265h53m28.476577712s) [noise=true , address=145.79.227.11:4732 , piece_id=3OSGIDEGOSVIMIBBTTRBMXJHGT626AC4GYDC3U2OY62SWB6FEV6A, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.605842790 (265h53m28.476925825s) [noise=true , address=45.91.116.8:4047 , piece_id=SOIENQXK3MCGBRBA6FOKCHEDEWIUMHHRUXO4FSPG2525XY4DXZCA, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.606015409 (265h53m28.477098454s) [noise=true , address=77.90.31.18:4808 , piece_id=QJT3MASV2WG5WBUSDLTETUHEL74QP4NWC5JFXVQQF5WYXY6DCOJA, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:28.606383579 (265h53m28.477466624s) [noise=true , address=45.91.116.5:4297 , piece_id=YT3FOYJEKBYTGDMA4Q2MKMJBLQOHQVPFZXSI322GZ7HGQDZ33ESA, seg_pos={0 0}, upload=233070]: Uploading piece... Dec 16 17:22:29.753134117 (265h53m29.624217162s) [noise=true , address=77.90.31.21:4877 , piece_id=MWYGIKFVXISWBUXVB4WEWBE3BOP4VCIOD6I3MBWBYCZEV7YCIHNQ, seg_pos={0 0}, upload=233070]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 16 17:22:30.332323516 (265h53m30.203406561s) [upload=233070]: Next returned nil segment Dec 16 17:22:30.332340469 (265h53m30.203423514s) [upload=233070]: Waiting for error group managing segments... Dec 16 17:22:30.616216770 (265h53m30.487299815s) [noise=true , address=145.79.227.35:4481 , piece_id=R5KOPMCY42A45GH2HEAIHRUXQORFVSJEYW7YAYU23HEDH6OUBQWQ, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.622252243 (265h53m30.493335278s) [noise=true , address=145.79.227.52:4446 , piece_id=4RN2ZBGTULZ6VQSO4H6VEKIDEG2DLMDUW74LMUXOZGPXNBWF7GHQ, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.623047456 (265h53m30.494130491s) [noise=true , address=145.79.227.46:4601 , piece_id=WV6ECF6MO4W65RNLWVHGCM4RC4I5E627UOTRQR6PTQ74ZJRIKD2A, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.633939096 (265h53m30.505022131s) [noise=true , address=145.79.227.27:4691 , piece_id=DT5BT7KS4FTRIGYLM3OQWHULUYOZUNEP6FZ7XUBB2RMORPPUM5ZQ, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.676237018 (265h53m30.547320063s) [noise=true , address=145.79.227.34:4907 , piece_id=C5GHGT5OGA7XRJIHJANVBAIF7Z6VMCDKI6WPEFTBDZPUKMIPLO5A, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.684076116 (265h53m30.555159151s) [noise=true , address=145.79.227.11:4732 , piece_id=3OSGIDEGOSVIMIBBTTRBMXJHGT626AC4GYDC3U2OY62SWB6FEV6A, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.851675636 (265h53m30.722758681s) [noise=true , address=77.90.31.11:4554 , piece_id=IMT5XXWJZL3LHQMWVMCIU7WPX2OGOFWQ6EH7HPOO3O4ANN6PVSIA, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.923476907 (265h53m30.794559942s) [noise=true , address=45.91.116.11:4118 , piece_id=MXVHHSHA7CHNU7BQZVFE52JRATWAUBKFMAW662FQPPDIKANVGK2Q, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.927061062 (265h53m30.798144107s) [noise=true , address=45.91.116.8:4047 , piece_id=SOIENQXK3MCGBRBA6FOKCHEDEWIUMHHRUXO4FSPG2525XY4DXZCA, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.940531031 (265h53m30.811614067s) [noise=true , address=45.91.116.14:4191 , piece_id=ELCLNIWDWE2STLW5U6FZL6OMIEMRGUSMEB4E3CTUX5V64RMXP2QQ, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:30.956255193 (265h53m30.827338238s) [noise=true , address=45.91.116.5:4297 , piece_id=YT3FOYJEKBYTGDMA4Q2MKMJBLQOHQVPFZXSI322GZ7HGQDZ33ESA, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:31.004934157 (265h53m30.876017202s) [noise=true , address=45.91.116.7:4009 , piece_id=54XGXE5QJ6UZJBV67K4DBDMPXG6A72PPSE3UQ3OXHU53RFP6KZ3A, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:31.126466316 (265h53m30.997549361s) [noise=true , address=45.91.116.13:4154 , piece_id=EWWYIQY4FLTYPK4GDOAQPT4CTSM4WZBIJB2TZILD65G5RD6LN4XA, seg_pos={0 0}, upload=233070]: Done uploading piece. err: <nil> Dec 16 17:22:31.126490482 (265h53m30.997573517s) [seg_pos={0 0}, upload=233070]: Segment reached optimal threshold of 13 pieces. Dec 16 17:22:31.126883510 (265h53m30.997966556s) [noise=true , address=45.91.116.10:4081 , piece_id=F6X4FPZ3PYRL2V6B5BTZLL36JSDIFA2AZUMB2Q3TEBVHDOPDGBRA, seg_pos={0 0}, upload=233070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:60060->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:60060->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60060->45.91.116.10:4081: use of closed network connection Dec 16 17:22:31.126900472 (265h53m30.997983517s) [noise=true , address=45.91.116.10:4081 , piece_id=F6X4FPZ3PYRL2V6B5BTZLL36JSDIFA2AZUMB2Q3TEBVHDOPDGBRA, seg_pos={0 0}, upload=233070]: Long tail cancellation due to optimal threshold Dec 16 17:22:31.126980125 (265h53m30.99806317s) [noise=true , address=77.90.31.18:4808 , piece_id=QJT3MASV2WG5WBUSDLTETUHEL74QP4NWC5JFXVQQF5WYXY6DCOJA, seg_pos={0 0}, upload=233070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:35310->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:35310->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35310->77.90.31.18:4808: use of closed network connection Dec 16 17:22:31.126991246 (265h53m30.998074291s) [noise=true , address=77.90.31.18:4808 , piece_id=QJT3MASV2WG5WBUSDLTETUHEL74QP4NWC5JFXVQQF5WYXY6DCOJA, seg_pos={0 0}, upload=233070]: Long tail cancellation due to optimal threshold Dec 16 17:22:31.127010171 (265h53m30.998093216s) [seg_pos={0 0}, upload=233070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:22:31.127091877 (265h53m30.998174912s) [upload=233070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:22:31.127098369 (265h53m30.998181404s) [upload=233070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:22:31.423343183 (265h53m31.294426218s) [upload=233070]: Done uploading segments... Dec 16 17:22:31.423362178 (265h53m31.294445213s) [upload=233070]: Upload finished. err: <nil> Duration: 5.803813734s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [X......X..X....X................XX....X...................XX] duration: 5.006765714s trace: https://jaeger.dev.storj.io/trace/2d7e37a5218d47ae FAIL (2025-12-16 17:23:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.006792295s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)