Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.....XX...................X...X......XX........X............] duration: 3.705218711s trace: https://jaeger.dev.storj.io/trace/42e4dc6b8c254bbe FAIL (2025-12-16 18:03:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.705259899s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X......X......X..........................................] duration: 6.012449321s trace: https://jaeger.dev.storj.io/trace/3a8b2eb64e78cdb7 FAIL (2025-12-16 18:03:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 17:33:25.825412431 (266h4m25.696495476s) [upload=233226]: Starting upload Dec 16 17:33:25.825716299 (266h4m25.696799334s) [upload=233226]: Done starting upload Dec 16 17:33:25.825900039 (266h4m25.696983074s) [upload=233226]: Uploading segments... Dec 16 17:33:25.825986874 (266h4m25.697069919s) [upload=233226]: Got next segment. Inline: false Dec 16 17:33:25.826024115 (266h4m25.69710716s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:33:25.826029405 (266h4m25.69711245s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:33:26.081528100 (266h4m25.952611145s) [seg_pos={0 0}, upload=233226]: Begin upload segment... Dec 16 17:33:26.081578937 (266h4m25.952661972s) [seg_pos={0 0}, upload=233226]: Done begin upload segment. Dec 16 17:33:26.081646916 (266h4m25.952729951s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.081689087 (266h4m25.952772122s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.082557751 (266h4m25.953640796s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083162883 (266h4m25.954245928s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083811628 (266h4m25.954894674s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.084268108 (266h4m25.955351153s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086136556 (266h4m25.957219591s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086764421 (266h4m25.957847456s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.087471016 (266h4m25.958554061s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.088841396 (266h4m25.959924442s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.089031668 (266h4m25.960114703s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.090373093 (266h4m25.961456128s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.092053242 (266h4m25.963136287s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.095858979 (266h4m25.966942024s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.096244182 (266h4m25.967327227s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.099898400 (266h4m25.970981445s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:27.793861758 (266h4m27.664944803s) [upload=233226]: Next returned nil segment Dec 16 17:33:27.793883430 (266h4m27.664966465s) [upload=233226]: Waiting for error group managing segments... Dec 16 17:33:28.076919891 (266h4m27.948002936s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.080532791 (266h4m27.951615826s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.081529298 (266h4m27.952612333s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.088220410 (266h4m27.959303455s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.102098585 (266h4m27.97318163s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.110557603 (266h4m27.981640648s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.112773352 (266h4m27.983856387s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.168421046 (266h4m28.039504091s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.175508041 (266h4m28.046591076s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.367477684 (266h4m28.23856073s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.388567654 (266h4m28.259650699s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.393640814 (266h4m28.264723859s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466522674 (266h4m28.337605719s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466547552 (266h4m28.337630597s) [seg_pos={0 0}, upload=233226]: Segment reached optimal threshold of 13 pieces. Dec 16 17:33:28.466920051 (266h4m28.338003096s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466932094 (266h4m28.338015129s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.466960198 (266h4m28.338043243s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466972481 (266h4m28.338055526s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467172281 (266h4m28.338255336s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection Dec 16 17:33:28.467182901 (266h4m28.338265946s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467216766 (266h4m28.338299811s) [seg_pos={0 0}, upload=233226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:33:28.467315314 (266h4m28.338398349s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:33:28.467322477 (266h4m28.338405522s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:33:28.799678512 (266h4m28.670761547s) [upload=233226]: Done uploading segments... Dec 16 17:33:28.799699542 (266h4m28.670782587s) [upload=233226]: Upload finished. err: <nil> Duration: 6.012472946s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.809591586s trace: https://jaeger.dev.storj.io/trace/2130d3e94d4b0821 FAIL (2025-12-16 18:02:58 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37318->45.38.24.2:443: read: connection reset by peer Duration: 2.809620512s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [......XX...................X...X......XX........X...........] duration: 3.612166703s trace: https://jaeger.dev.storj.io/trace/42e4dc6b8c254bbe FAIL (2025-12-16 18:02:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.612194246s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X......X......X.........................................] duration: 5.896216604s trace: https://jaeger.dev.storj.io/trace/3a8b2eb64e78cdb7 FAIL (2025-12-16 18:02:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 17:33:25.825412431 (266h4m25.696495476s) [upload=233226]: Starting upload Dec 16 17:33:25.825716299 (266h4m25.696799334s) [upload=233226]: Done starting upload Dec 16 17:33:25.825900039 (266h4m25.696983074s) [upload=233226]: Uploading segments... Dec 16 17:33:25.825986874 (266h4m25.697069919s) [upload=233226]: Got next segment. Inline: false Dec 16 17:33:25.826024115 (266h4m25.69710716s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:33:25.826029405 (266h4m25.69711245s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:33:26.081528100 (266h4m25.952611145s) [seg_pos={0 0}, upload=233226]: Begin upload segment... Dec 16 17:33:26.081578937 (266h4m25.952661972s) [seg_pos={0 0}, upload=233226]: Done begin upload segment. Dec 16 17:33:26.081646916 (266h4m25.952729951s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.081689087 (266h4m25.952772122s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.082557751 (266h4m25.953640796s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083162883 (266h4m25.954245928s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083811628 (266h4m25.954894674s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.084268108 (266h4m25.955351153s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086136556 (266h4m25.957219591s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086764421 (266h4m25.957847456s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.087471016 (266h4m25.958554061s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.088841396 (266h4m25.959924442s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.089031668 (266h4m25.960114703s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.090373093 (266h4m25.961456128s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.092053242 (266h4m25.963136287s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.095858979 (266h4m25.966942024s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.096244182 (266h4m25.967327227s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.099898400 (266h4m25.970981445s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:27.793861758 (266h4m27.664944803s) [upload=233226]: Next returned nil segment Dec 16 17:33:27.793883430 (266h4m27.664966465s) [upload=233226]: Waiting for error group managing segments... Dec 16 17:33:28.076919891 (266h4m27.948002936s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.080532791 (266h4m27.951615826s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.081529298 (266h4m27.952612333s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.088220410 (266h4m27.959303455s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.102098585 (266h4m27.97318163s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.110557603 (266h4m27.981640648s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.112773352 (266h4m27.983856387s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.168421046 (266h4m28.039504091s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.175508041 (266h4m28.046591076s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.367477684 (266h4m28.23856073s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.388567654 (266h4m28.259650699s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.393640814 (266h4m28.264723859s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466522674 (266h4m28.337605719s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466547552 (266h4m28.337630597s) [seg_pos={0 0}, upload=233226]: Segment reached optimal threshold of 13 pieces. Dec 16 17:33:28.466920051 (266h4m28.338003096s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466932094 (266h4m28.338015129s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.466960198 (266h4m28.338043243s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466972481 (266h4m28.338055526s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467172281 (266h4m28.338255336s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection Dec 16 17:33:28.467182901 (266h4m28.338265946s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467216766 (266h4m28.338299811s) [seg_pos={0 0}, upload=233226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:33:28.467315314 (266h4m28.338398349s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:33:28.467322477 (266h4m28.338405522s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:33:28.799678512 (266h4m28.670761547s) [upload=233226]: Done uploading segments... Dec 16 17:33:28.799699542 (266h4m28.670782587s) [upload=233226]: Upload finished. err: <nil> Duration: 5.896244307s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.......XX...................X...X......XX........X..........] duration: 4.005295039s trace: https://jaeger.dev.storj.io/trace/42e4dc6b8c254bbe FAIL (2025-12-16 18:02:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.005319346s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X......X......X........................................] duration: 6.001207584s trace: https://jaeger.dev.storj.io/trace/3a8b2eb64e78cdb7 FAIL (2025-12-16 18:02:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 17:33:25.825412431 (266h4m25.696495476s) [upload=233226]: Starting upload Dec 16 17:33:25.825716299 (266h4m25.696799334s) [upload=233226]: Done starting upload Dec 16 17:33:25.825900039 (266h4m25.696983074s) [upload=233226]: Uploading segments... Dec 16 17:33:25.825986874 (266h4m25.697069919s) [upload=233226]: Got next segment. Inline: false Dec 16 17:33:25.826024115 (266h4m25.69710716s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:33:25.826029405 (266h4m25.69711245s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:33:26.081528100 (266h4m25.952611145s) [seg_pos={0 0}, upload=233226]: Begin upload segment... Dec 16 17:33:26.081578937 (266h4m25.952661972s) [seg_pos={0 0}, upload=233226]: Done begin upload segment. Dec 16 17:33:26.081646916 (266h4m25.952729951s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.081689087 (266h4m25.952772122s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.082557751 (266h4m25.953640796s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083162883 (266h4m25.954245928s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083811628 (266h4m25.954894674s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.084268108 (266h4m25.955351153s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086136556 (266h4m25.957219591s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086764421 (266h4m25.957847456s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.087471016 (266h4m25.958554061s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.088841396 (266h4m25.959924442s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.089031668 (266h4m25.960114703s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.090373093 (266h4m25.961456128s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.092053242 (266h4m25.963136287s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.095858979 (266h4m25.966942024s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.096244182 (266h4m25.967327227s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.099898400 (266h4m25.970981445s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:27.793861758 (266h4m27.664944803s) [upload=233226]: Next returned nil segment Dec 16 17:33:27.793883430 (266h4m27.664966465s) [upload=233226]: Waiting for error group managing segments... Dec 16 17:33:28.076919891 (266h4m27.948002936s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.080532791 (266h4m27.951615826s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.081529298 (266h4m27.952612333s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.088220410 (266h4m27.959303455s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.102098585 (266h4m27.97318163s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.110557603 (266h4m27.981640648s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.112773352 (266h4m27.983856387s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.168421046 (266h4m28.039504091s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.175508041 (266h4m28.046591076s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.367477684 (266h4m28.23856073s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.388567654 (266h4m28.259650699s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.393640814 (266h4m28.264723859s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466522674 (266h4m28.337605719s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466547552 (266h4m28.337630597s) [seg_pos={0 0}, upload=233226]: Segment reached optimal threshold of 13 pieces. Dec 16 17:33:28.466920051 (266h4m28.338003096s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466932094 (266h4m28.338015129s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.466960198 (266h4m28.338043243s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466972481 (266h4m28.338055526s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467172281 (266h4m28.338255336s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection Dec 16 17:33:28.467182901 (266h4m28.338265946s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467216766 (266h4m28.338299811s) [seg_pos={0 0}, upload=233226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:33:28.467315314 (266h4m28.338398349s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:33:28.467322477 (266h4m28.338405522s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:33:28.799678512 (266h4m28.670761547s) [upload=233226]: Done uploading segments... Dec 16 17:33:28.799699542 (266h4m28.670782587s) [upload=233226]: Upload finished. err: <nil> Duration: 6.001240637s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X......X......X.......................................] duration: 5.982466809s trace: https://jaeger.dev.storj.io/trace/3a8b2eb64e78cdb7 FAIL (2025-12-16 18:01:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 17:33:25.825412431 (266h4m25.696495476s) [upload=233226]: Starting upload Dec 16 17:33:25.825716299 (266h4m25.696799334s) [upload=233226]: Done starting upload Dec 16 17:33:25.825900039 (266h4m25.696983074s) [upload=233226]: Uploading segments... Dec 16 17:33:25.825986874 (266h4m25.697069919s) [upload=233226]: Got next segment. Inline: false Dec 16 17:33:25.826024115 (266h4m25.69710716s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:33:25.826029405 (266h4m25.69711245s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:33:26.081528100 (266h4m25.952611145s) [seg_pos={0 0}, upload=233226]: Begin upload segment... Dec 16 17:33:26.081578937 (266h4m25.952661972s) [seg_pos={0 0}, upload=233226]: Done begin upload segment. Dec 16 17:33:26.081646916 (266h4m25.952729951s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.081689087 (266h4m25.952772122s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.082557751 (266h4m25.953640796s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083162883 (266h4m25.954245928s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083811628 (266h4m25.954894674s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.084268108 (266h4m25.955351153s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086136556 (266h4m25.957219591s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086764421 (266h4m25.957847456s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.087471016 (266h4m25.958554061s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.088841396 (266h4m25.959924442s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.089031668 (266h4m25.960114703s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.090373093 (266h4m25.961456128s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.092053242 (266h4m25.963136287s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.095858979 (266h4m25.966942024s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.096244182 (266h4m25.967327227s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.099898400 (266h4m25.970981445s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:27.793861758 (266h4m27.664944803s) [upload=233226]: Next returned nil segment Dec 16 17:33:27.793883430 (266h4m27.664966465s) [upload=233226]: Waiting for error group managing segments... Dec 16 17:33:28.076919891 (266h4m27.948002936s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.080532791 (266h4m27.951615826s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.081529298 (266h4m27.952612333s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.088220410 (266h4m27.959303455s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.102098585 (266h4m27.97318163s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.110557603 (266h4m27.981640648s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.112773352 (266h4m27.983856387s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.168421046 (266h4m28.039504091s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.175508041 (266h4m28.046591076s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.367477684 (266h4m28.23856073s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.388567654 (266h4m28.259650699s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.393640814 (266h4m28.264723859s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466522674 (266h4m28.337605719s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466547552 (266h4m28.337630597s) [seg_pos={0 0}, upload=233226]: Segment reached optimal threshold of 13 pieces. Dec 16 17:33:28.466920051 (266h4m28.338003096s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466932094 (266h4m28.338015129s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.466960198 (266h4m28.338043243s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466972481 (266h4m28.338055526s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467172281 (266h4m28.338255336s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection Dec 16 17:33:28.467182901 (266h4m28.338265946s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467216766 (266h4m28.338299811s) [seg_pos={0 0}, upload=233226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:33:28.467315314 (266h4m28.338398349s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:33:28.467322477 (266h4m28.338405522s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:33:28.799678512 (266h4m28.670761547s) [upload=233226]: Done uploading segments... Dec 16 17:33:28.799699542 (266h4m28.670782587s) [upload=233226]: Upload finished. err: <nil> Duration: 5.982496175s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.416992248s trace: https://jaeger.dev.storj.io/trace/2130d3e94d4b0821 FAIL (2025-12-16 18:01:52 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37318->45.38.24.2:443: read: connection reset by peer Duration: 2.417023457s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X......X......X......................................] duration: 6.057914336s trace: https://jaeger.dev.storj.io/trace/3a8b2eb64e78cdb7 FAIL (2025-12-16 18:01:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 17:33:25.825412431 (266h4m25.696495476s) [upload=233226]: Starting upload Dec 16 17:33:25.825716299 (266h4m25.696799334s) [upload=233226]: Done starting upload Dec 16 17:33:25.825900039 (266h4m25.696983074s) [upload=233226]: Uploading segments... Dec 16 17:33:25.825986874 (266h4m25.697069919s) [upload=233226]: Got next segment. Inline: false Dec 16 17:33:25.826024115 (266h4m25.69710716s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:33:25.826029405 (266h4m25.69711245s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:33:26.081528100 (266h4m25.952611145s) [seg_pos={0 0}, upload=233226]: Begin upload segment... Dec 16 17:33:26.081578937 (266h4m25.952661972s) [seg_pos={0 0}, upload=233226]: Done begin upload segment. Dec 16 17:33:26.081646916 (266h4m25.952729951s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.081689087 (266h4m25.952772122s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.082557751 (266h4m25.953640796s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083162883 (266h4m25.954245928s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083811628 (266h4m25.954894674s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.084268108 (266h4m25.955351153s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086136556 (266h4m25.957219591s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086764421 (266h4m25.957847456s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.087471016 (266h4m25.958554061s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.088841396 (266h4m25.959924442s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.089031668 (266h4m25.960114703s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.090373093 (266h4m25.961456128s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.092053242 (266h4m25.963136287s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.095858979 (266h4m25.966942024s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.096244182 (266h4m25.967327227s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.099898400 (266h4m25.970981445s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:27.793861758 (266h4m27.664944803s) [upload=233226]: Next returned nil segment Dec 16 17:33:27.793883430 (266h4m27.664966465s) [upload=233226]: Waiting for error group managing segments... Dec 16 17:33:28.076919891 (266h4m27.948002936s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.080532791 (266h4m27.951615826s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.081529298 (266h4m27.952612333s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.088220410 (266h4m27.959303455s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.102098585 (266h4m27.97318163s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.110557603 (266h4m27.981640648s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.112773352 (266h4m27.983856387s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.168421046 (266h4m28.039504091s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.175508041 (266h4m28.046591076s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.367477684 (266h4m28.23856073s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.388567654 (266h4m28.259650699s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.393640814 (266h4m28.264723859s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466522674 (266h4m28.337605719s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466547552 (266h4m28.337630597s) [seg_pos={0 0}, upload=233226]: Segment reached optimal threshold of 13 pieces. Dec 16 17:33:28.466920051 (266h4m28.338003096s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466932094 (266h4m28.338015129s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.466960198 (266h4m28.338043243s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466972481 (266h4m28.338055526s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467172281 (266h4m28.338255336s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection Dec 16 17:33:28.467182901 (266h4m28.338265946s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467216766 (266h4m28.338299811s) [seg_pos={0 0}, upload=233226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:33:28.467315314 (266h4m28.338398349s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:33:28.467322477 (266h4m28.338405522s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:33:28.799678512 (266h4m28.670761547s) [upload=233226]: Done uploading segments... Dec 16 17:33:28.799699542 (266h4m28.670782587s) [upload=233226]: Upload finished. err: <nil> Duration: 6.057947378s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [........XX...................X...X......XX........X.........] duration: 3.732543221s trace: https://jaeger.dev.storj.io/trace/42e4dc6b8c254bbe FAIL (2025-12-16 18:01:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.732575332s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X......X......X.....................................] duration: 5.940904507s trace: https://jaeger.dev.storj.io/trace/3a8b2eb64e78cdb7 FAIL (2025-12-16 18:00:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 17:33:25.825412431 (266h4m25.696495476s) [upload=233226]: Starting upload Dec 16 17:33:25.825716299 (266h4m25.696799334s) [upload=233226]: Done starting upload Dec 16 17:33:25.825900039 (266h4m25.696983074s) [upload=233226]: Uploading segments... Dec 16 17:33:25.825986874 (266h4m25.697069919s) [upload=233226]: Got next segment. Inline: false Dec 16 17:33:25.826024115 (266h4m25.69710716s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:33:25.826029405 (266h4m25.69711245s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:33:26.081528100 (266h4m25.952611145s) [seg_pos={0 0}, upload=233226]: Begin upload segment... Dec 16 17:33:26.081578937 (266h4m25.952661972s) [seg_pos={0 0}, upload=233226]: Done begin upload segment. Dec 16 17:33:26.081646916 (266h4m25.952729951s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.081689087 (266h4m25.952772122s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.082557751 (266h4m25.953640796s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083162883 (266h4m25.954245928s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083811628 (266h4m25.954894674s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.084268108 (266h4m25.955351153s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086136556 (266h4m25.957219591s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086764421 (266h4m25.957847456s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.087471016 (266h4m25.958554061s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.088841396 (266h4m25.959924442s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.089031668 (266h4m25.960114703s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.090373093 (266h4m25.961456128s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.092053242 (266h4m25.963136287s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.095858979 (266h4m25.966942024s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.096244182 (266h4m25.967327227s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.099898400 (266h4m25.970981445s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:27.793861758 (266h4m27.664944803s) [upload=233226]: Next returned nil segment Dec 16 17:33:27.793883430 (266h4m27.664966465s) [upload=233226]: Waiting for error group managing segments... Dec 16 17:33:28.076919891 (266h4m27.948002936s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.080532791 (266h4m27.951615826s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.081529298 (266h4m27.952612333s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.088220410 (266h4m27.959303455s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.102098585 (266h4m27.97318163s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.110557603 (266h4m27.981640648s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.112773352 (266h4m27.983856387s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.168421046 (266h4m28.039504091s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.175508041 (266h4m28.046591076s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.367477684 (266h4m28.23856073s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.388567654 (266h4m28.259650699s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.393640814 (266h4m28.264723859s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466522674 (266h4m28.337605719s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466547552 (266h4m28.337630597s) [seg_pos={0 0}, upload=233226]: Segment reached optimal threshold of 13 pieces. Dec 16 17:33:28.466920051 (266h4m28.338003096s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466932094 (266h4m28.338015129s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.466960198 (266h4m28.338043243s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466972481 (266h4m28.338055526s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467172281 (266h4m28.338255336s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection Dec 16 17:33:28.467182901 (266h4m28.338265946s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467216766 (266h4m28.338299811s) [seg_pos={0 0}, upload=233226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:33:28.467315314 (266h4m28.338398349s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:33:28.467322477 (266h4m28.338405522s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:33:28.799678512 (266h4m28.670761547s) [upload=233226]: Done uploading segments... Dec 16 17:33:28.799699542 (266h4m28.670782587s) [upload=233226]: Upload finished. err: <nil> Duration: 5.940932501s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.366920432s trace: https://jaeger.dev.storj.io/trace/2130d3e94d4b0821 FAIL (2025-12-16 18:00:47 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37318->45.38.24.2:443: read: connection reset by peer Duration: 2.366948715s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.........XX...................X...X......XX........X........] duration: 4.58905768s trace: https://jaeger.dev.storj.io/trace/42e4dc6b8c254bbe FAIL (2025-12-16 18:00:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.589089872s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.47296441s trace: https://jaeger.dev.storj.io/trace/2130d3e94d4b0821 FAIL (2025-12-16 18:00:16 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37318->45.38.24.2:443: read: connection reset by peer Duration: 2.473107893s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X......X......X....................................] duration: 18.445675183s trace: https://jaeger.dev.storj.io/trace/3a8b2eb64e78cdb7 FAIL (2025-12-16 18:00:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 17:33:25.825412431 (266h4m25.696495476s) [upload=233226]: Starting upload Dec 16 17:33:25.825716299 (266h4m25.696799334s) [upload=233226]: Done starting upload Dec 16 17:33:25.825900039 (266h4m25.696983074s) [upload=233226]: Uploading segments... Dec 16 17:33:25.825986874 (266h4m25.697069919s) [upload=233226]: Got next segment. Inline: false Dec 16 17:33:25.826024115 (266h4m25.69710716s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:33:25.826029405 (266h4m25.69711245s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:33:26.081528100 (266h4m25.952611145s) [seg_pos={0 0}, upload=233226]: Begin upload segment... Dec 16 17:33:26.081578937 (266h4m25.952661972s) [seg_pos={0 0}, upload=233226]: Done begin upload segment. Dec 16 17:33:26.081646916 (266h4m25.952729951s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.081689087 (266h4m25.952772122s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.082557751 (266h4m25.953640796s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083162883 (266h4m25.954245928s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083811628 (266h4m25.954894674s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.084268108 (266h4m25.955351153s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086136556 (266h4m25.957219591s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086764421 (266h4m25.957847456s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.087471016 (266h4m25.958554061s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.088841396 (266h4m25.959924442s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.089031668 (266h4m25.960114703s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.090373093 (266h4m25.961456128s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.092053242 (266h4m25.963136287s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.095858979 (266h4m25.966942024s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.096244182 (266h4m25.967327227s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.099898400 (266h4m25.970981445s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:27.793861758 (266h4m27.664944803s) [upload=233226]: Next returned nil segment Dec 16 17:33:27.793883430 (266h4m27.664966465s) [upload=233226]: Waiting for error group managing segments... Dec 16 17:33:28.076919891 (266h4m27.948002936s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.080532791 (266h4m27.951615826s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.081529298 (266h4m27.952612333s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.088220410 (266h4m27.959303455s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.102098585 (266h4m27.97318163s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.110557603 (266h4m27.981640648s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.112773352 (266h4m27.983856387s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.168421046 (266h4m28.039504091s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.175508041 (266h4m28.046591076s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.367477684 (266h4m28.23856073s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.388567654 (266h4m28.259650699s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.393640814 (266h4m28.264723859s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466522674 (266h4m28.337605719s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466547552 (266h4m28.337630597s) [seg_pos={0 0}, upload=233226]: Segment reached optimal threshold of 13 pieces. Dec 16 17:33:28.466920051 (266h4m28.338003096s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466932094 (266h4m28.338015129s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.466960198 (266h4m28.338043243s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466972481 (266h4m28.338055526s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467172281 (266h4m28.338255336s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection Dec 16 17:33:28.467182901 (266h4m28.338265946s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467216766 (266h4m28.338299811s) [seg_pos={0 0}, upload=233226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:33:28.467315314 (266h4m28.338398349s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:33:28.467322477 (266h4m28.338405522s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:33:28.799678512 (266h4m28.670761547s) [upload=233226]: Done uploading segments... Dec 16 17:33:28.799699542 (266h4m28.670782587s) [upload=233226]: Upload finished. err: <nil> Duration: 18.445709448s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..........XX...................X...X......XX........X.......] duration: 3.765501891s trace: https://jaeger.dev.storj.io/trace/42e4dc6b8c254bbe FAIL (2025-12-16 17:59:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.765520355s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.390836531s trace: https://jaeger.dev.storj.io/trace/2130d3e94d4b0821 FAIL (2025-12-16 17:59:11 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37318->45.38.24.2:443: read: connection reset by peer Duration: 2.390879413s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X......X......X...................................] duration: 5.974703688s trace: https://jaeger.dev.storj.io/trace/3a8b2eb64e78cdb7 FAIL (2025-12-16 17:59:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 17:33:25.825412431 (266h4m25.696495476s) [upload=233226]: Starting upload Dec 16 17:33:25.825716299 (266h4m25.696799334s) [upload=233226]: Done starting upload Dec 16 17:33:25.825900039 (266h4m25.696983074s) [upload=233226]: Uploading segments... Dec 16 17:33:25.825986874 (266h4m25.697069919s) [upload=233226]: Got next segment. Inline: false Dec 16 17:33:25.826024115 (266h4m25.69710716s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:33:25.826029405 (266h4m25.69711245s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:33:26.081528100 (266h4m25.952611145s) [seg_pos={0 0}, upload=233226]: Begin upload segment... Dec 16 17:33:26.081578937 (266h4m25.952661972s) [seg_pos={0 0}, upload=233226]: Done begin upload segment. Dec 16 17:33:26.081646916 (266h4m25.952729951s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.081689087 (266h4m25.952772122s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.082557751 (266h4m25.953640796s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083162883 (266h4m25.954245928s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083811628 (266h4m25.954894674s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.084268108 (266h4m25.955351153s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086136556 (266h4m25.957219591s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086764421 (266h4m25.957847456s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.087471016 (266h4m25.958554061s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.088841396 (266h4m25.959924442s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.089031668 (266h4m25.960114703s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.090373093 (266h4m25.961456128s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.092053242 (266h4m25.963136287s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.095858979 (266h4m25.966942024s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.096244182 (266h4m25.967327227s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.099898400 (266h4m25.970981445s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:27.793861758 (266h4m27.664944803s) [upload=233226]: Next returned nil segment Dec 16 17:33:27.793883430 (266h4m27.664966465s) [upload=233226]: Waiting for error group managing segments... Dec 16 17:33:28.076919891 (266h4m27.948002936s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.080532791 (266h4m27.951615826s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.081529298 (266h4m27.952612333s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.088220410 (266h4m27.959303455s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.102098585 (266h4m27.97318163s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.110557603 (266h4m27.981640648s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.112773352 (266h4m27.983856387s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.168421046 (266h4m28.039504091s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.175508041 (266h4m28.046591076s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.367477684 (266h4m28.23856073s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.388567654 (266h4m28.259650699s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.393640814 (266h4m28.264723859s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466522674 (266h4m28.337605719s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466547552 (266h4m28.337630597s) [seg_pos={0 0}, upload=233226]: Segment reached optimal threshold of 13 pieces. Dec 16 17:33:28.466920051 (266h4m28.338003096s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466932094 (266h4m28.338015129s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.466960198 (266h4m28.338043243s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466972481 (266h4m28.338055526s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467172281 (266h4m28.338255336s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection Dec 16 17:33:28.467182901 (266h4m28.338265946s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467216766 (266h4m28.338299811s) [seg_pos={0 0}, upload=233226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:33:28.467315314 (266h4m28.338398349s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:33:28.467322477 (266h4m28.338405522s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:33:28.799678512 (266h4m28.670761547s) [upload=233226]: Done uploading segments... Dec 16 17:33:28.799699542 (266h4m28.670782587s) [upload=233226]: Upload finished. err: <nil> Duration: 5.97473681s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [...........XX...................X...X......XX........X......] duration: 3.948943557s trace: https://jaeger.dev.storj.io/trace/42e4dc6b8c254bbe FAIL (2025-12-16 17:59:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.948985166s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [............XX...................X...X......XX........X.....] duration: 3.526522872s trace: https://jaeger.dev.storj.io/trace/42e4dc6b8c254bbe FAIL (2025-12-16 17:58:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.526547228s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.323745357s trace: https://jaeger.dev.storj.io/trace/2130d3e94d4b0821 FAIL (2025-12-16 17:58:06 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37318->45.38.24.2:443: read: connection reset by peer Duration: 2.323837863s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X......X......X..................................] duration: 5.872110361s trace: https://jaeger.dev.storj.io/trace/3a8b2eb64e78cdb7 FAIL (2025-12-16 17:57:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 17:33:25.825412431 (266h4m25.696495476s) [upload=233226]: Starting upload Dec 16 17:33:25.825716299 (266h4m25.696799334s) [upload=233226]: Done starting upload Dec 16 17:33:25.825900039 (266h4m25.696983074s) [upload=233226]: Uploading segments... Dec 16 17:33:25.825986874 (266h4m25.697069919s) [upload=233226]: Got next segment. Inline: false Dec 16 17:33:25.826024115 (266h4m25.69710716s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:33:25.826029405 (266h4m25.69711245s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:33:26.081528100 (266h4m25.952611145s) [seg_pos={0 0}, upload=233226]: Begin upload segment... Dec 16 17:33:26.081578937 (266h4m25.952661972s) [seg_pos={0 0}, upload=233226]: Done begin upload segment. Dec 16 17:33:26.081646916 (266h4m25.952729951s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.081689087 (266h4m25.952772122s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.082557751 (266h4m25.953640796s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083162883 (266h4m25.954245928s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083811628 (266h4m25.954894674s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.084268108 (266h4m25.955351153s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086136556 (266h4m25.957219591s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086764421 (266h4m25.957847456s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.087471016 (266h4m25.958554061s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.088841396 (266h4m25.959924442s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.089031668 (266h4m25.960114703s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.090373093 (266h4m25.961456128s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.092053242 (266h4m25.963136287s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.095858979 (266h4m25.966942024s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.096244182 (266h4m25.967327227s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.099898400 (266h4m25.970981445s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:27.793861758 (266h4m27.664944803s) [upload=233226]: Next returned nil segment Dec 16 17:33:27.793883430 (266h4m27.664966465s) [upload=233226]: Waiting for error group managing segments... Dec 16 17:33:28.076919891 (266h4m27.948002936s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.080532791 (266h4m27.951615826s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.081529298 (266h4m27.952612333s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.088220410 (266h4m27.959303455s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.102098585 (266h4m27.97318163s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.110557603 (266h4m27.981640648s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.112773352 (266h4m27.983856387s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.168421046 (266h4m28.039504091s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.175508041 (266h4m28.046591076s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.367477684 (266h4m28.23856073s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.388567654 (266h4m28.259650699s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.393640814 (266h4m28.264723859s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466522674 (266h4m28.337605719s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466547552 (266h4m28.337630597s) [seg_pos={0 0}, upload=233226]: Segment reached optimal threshold of 13 pieces. Dec 16 17:33:28.466920051 (266h4m28.338003096s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466932094 (266h4m28.338015129s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.466960198 (266h4m28.338043243s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466972481 (266h4m28.338055526s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467172281 (266h4m28.338255336s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection Dec 16 17:33:28.467182901 (266h4m28.338265946s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467216766 (266h4m28.338299811s) [seg_pos={0 0}, upload=233226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:33:28.467315314 (266h4m28.338398349s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:33:28.467322477 (266h4m28.338405522s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:33:28.799678512 (266h4m28.670761547s) [upload=233226]: Done uploading segments... Dec 16 17:33:28.799699542 (266h4m28.670782587s) [upload=233226]: Upload finished. err: <nil> Duration: 5.872142563s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.872792512s trace: https://jaeger.dev.storj.io/trace/2130d3e94d4b0821 FAIL (2025-12-16 17:57:34 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37318->45.38.24.2:443: read: connection reset by peer Duration: 2.872816489s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.............XX...................X...X......XX........X....] duration: 3.583774888s trace: https://jaeger.dev.storj.io/trace/42e4dc6b8c254bbe FAIL (2025-12-16 17:57:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.583805206s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X......X......X.................................] duration: 6.313401328s trace: https://jaeger.dev.storj.io/trace/3a8b2eb64e78cdb7 FAIL (2025-12-16 17:57:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 17:33:25.825412431 (266h4m25.696495476s) [upload=233226]: Starting upload Dec 16 17:33:25.825716299 (266h4m25.696799334s) [upload=233226]: Done starting upload Dec 16 17:33:25.825900039 (266h4m25.696983074s) [upload=233226]: Uploading segments... Dec 16 17:33:25.825986874 (266h4m25.697069919s) [upload=233226]: Got next segment. Inline: false Dec 16 17:33:25.826024115 (266h4m25.69710716s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:33:25.826029405 (266h4m25.69711245s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:33:26.081528100 (266h4m25.952611145s) [seg_pos={0 0}, upload=233226]: Begin upload segment... Dec 16 17:33:26.081578937 (266h4m25.952661972s) [seg_pos={0 0}, upload=233226]: Done begin upload segment. Dec 16 17:33:26.081646916 (266h4m25.952729951s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.081689087 (266h4m25.952772122s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.082557751 (266h4m25.953640796s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083162883 (266h4m25.954245928s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083811628 (266h4m25.954894674s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.084268108 (266h4m25.955351153s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086136556 (266h4m25.957219591s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086764421 (266h4m25.957847456s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.087471016 (266h4m25.958554061s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.088841396 (266h4m25.959924442s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.089031668 (266h4m25.960114703s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.090373093 (266h4m25.961456128s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.092053242 (266h4m25.963136287s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.095858979 (266h4m25.966942024s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.096244182 (266h4m25.967327227s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.099898400 (266h4m25.970981445s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:27.793861758 (266h4m27.664944803s) [upload=233226]: Next returned nil segment Dec 16 17:33:27.793883430 (266h4m27.664966465s) [upload=233226]: Waiting for error group managing segments... Dec 16 17:33:28.076919891 (266h4m27.948002936s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.080532791 (266h4m27.951615826s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.081529298 (266h4m27.952612333s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.088220410 (266h4m27.959303455s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.102098585 (266h4m27.97318163s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.110557603 (266h4m27.981640648s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.112773352 (266h4m27.983856387s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.168421046 (266h4m28.039504091s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.175508041 (266h4m28.046591076s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.367477684 (266h4m28.23856073s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.388567654 (266h4m28.259650699s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.393640814 (266h4m28.264723859s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466522674 (266h4m28.337605719s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466547552 (266h4m28.337630597s) [seg_pos={0 0}, upload=233226]: Segment reached optimal threshold of 13 pieces. Dec 16 17:33:28.466920051 (266h4m28.338003096s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466932094 (266h4m28.338015129s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.466960198 (266h4m28.338043243s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466972481 (266h4m28.338055526s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467172281 (266h4m28.338255336s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection Dec 16 17:33:28.467182901 (266h4m28.338265946s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467216766 (266h4m28.338299811s) [seg_pos={0 0}, upload=233226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:33:28.467315314 (266h4m28.338398349s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:33:28.467322477 (266h4m28.338405522s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:33:28.799678512 (266h4m28.670761547s) [upload=233226]: Done uploading segments... Dec 16 17:33:28.799699542 (266h4m28.670782587s) [upload=233226]: Upload finished. err: <nil> Duration: 6.313430763s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 4.099019267s trace: https://jaeger.dev.storj.io/trace/2130d3e94d4b0821 FAIL (2025-12-16 17:56:56 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37318->45.38.24.2:443: read: connection reset by peer Duration: 4.099041149s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X......X......X................................] duration: 5.664721615s trace: https://jaeger.dev.storj.io/trace/3a8b2eb64e78cdb7 FAIL (2025-12-16 17:56:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 17:33:25.825412431 (266h4m25.696495476s) [upload=233226]: Starting upload Dec 16 17:33:25.825716299 (266h4m25.696799334s) [upload=233226]: Done starting upload Dec 16 17:33:25.825900039 (266h4m25.696983074s) [upload=233226]: Uploading segments... Dec 16 17:33:25.825986874 (266h4m25.697069919s) [upload=233226]: Got next segment. Inline: false Dec 16 17:33:25.826024115 (266h4m25.69710716s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:33:25.826029405 (266h4m25.69711245s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:33:26.081528100 (266h4m25.952611145s) [seg_pos={0 0}, upload=233226]: Begin upload segment... Dec 16 17:33:26.081578937 (266h4m25.952661972s) [seg_pos={0 0}, upload=233226]: Done begin upload segment. Dec 16 17:33:26.081646916 (266h4m25.952729951s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.081689087 (266h4m25.952772122s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.082557751 (266h4m25.953640796s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083162883 (266h4m25.954245928s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083811628 (266h4m25.954894674s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.084268108 (266h4m25.955351153s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086136556 (266h4m25.957219591s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086764421 (266h4m25.957847456s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.087471016 (266h4m25.958554061s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.088841396 (266h4m25.959924442s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.089031668 (266h4m25.960114703s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.090373093 (266h4m25.961456128s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.092053242 (266h4m25.963136287s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.095858979 (266h4m25.966942024s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.096244182 (266h4m25.967327227s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.099898400 (266h4m25.970981445s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:27.793861758 (266h4m27.664944803s) [upload=233226]: Next returned nil segment Dec 16 17:33:27.793883430 (266h4m27.664966465s) [upload=233226]: Waiting for error group managing segments... Dec 16 17:33:28.076919891 (266h4m27.948002936s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.080532791 (266h4m27.951615826s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.081529298 (266h4m27.952612333s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.088220410 (266h4m27.959303455s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.102098585 (266h4m27.97318163s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.110557603 (266h4m27.981640648s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.112773352 (266h4m27.983856387s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.168421046 (266h4m28.039504091s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.175508041 (266h4m28.046591076s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.367477684 (266h4m28.23856073s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.388567654 (266h4m28.259650699s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.393640814 (266h4m28.264723859s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466522674 (266h4m28.337605719s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466547552 (266h4m28.337630597s) [seg_pos={0 0}, upload=233226]: Segment reached optimal threshold of 13 pieces. Dec 16 17:33:28.466920051 (266h4m28.338003096s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466932094 (266h4m28.338015129s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.466960198 (266h4m28.338043243s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466972481 (266h4m28.338055526s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467172281 (266h4m28.338255336s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection Dec 16 17:33:28.467182901 (266h4m28.338265946s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467216766 (266h4m28.338299811s) [seg_pos={0 0}, upload=233226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:33:28.467315314 (266h4m28.338398349s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:33:28.467322477 (266h4m28.338405522s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:33:28.799678512 (266h4m28.670761547s) [upload=233226]: Done uploading segments... Dec 16 17:33:28.799699542 (266h4m28.670782587s) [upload=233226]: Upload finished. err: <nil> Duration: 5.664766671s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..............XX...................X...X......XX........X...] duration: 3.619240101s trace: https://jaeger.dev.storj.io/trace/42e4dc6b8c254bbe FAIL (2025-12-16 17:56:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.61926509s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X......X......X...............................] duration: 5.675094938s trace: https://jaeger.dev.storj.io/trace/3a8b2eb64e78cdb7 FAIL (2025-12-16 17:56:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 17:33:25.825412431 (266h4m25.696495476s) [upload=233226]: Starting upload Dec 16 17:33:25.825716299 (266h4m25.696799334s) [upload=233226]: Done starting upload Dec 16 17:33:25.825900039 (266h4m25.696983074s) [upload=233226]: Uploading segments... Dec 16 17:33:25.825986874 (266h4m25.697069919s) [upload=233226]: Got next segment. Inline: false Dec 16 17:33:25.826024115 (266h4m25.69710716s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:33:25.826029405 (266h4m25.69711245s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:33:26.081528100 (266h4m25.952611145s) [seg_pos={0 0}, upload=233226]: Begin upload segment... Dec 16 17:33:26.081578937 (266h4m25.952661972s) [seg_pos={0 0}, upload=233226]: Done begin upload segment. Dec 16 17:33:26.081646916 (266h4m25.952729951s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.081689087 (266h4m25.952772122s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.082557751 (266h4m25.953640796s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083162883 (266h4m25.954245928s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083811628 (266h4m25.954894674s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.084268108 (266h4m25.955351153s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086136556 (266h4m25.957219591s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086764421 (266h4m25.957847456s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.087471016 (266h4m25.958554061s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.088841396 (266h4m25.959924442s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.089031668 (266h4m25.960114703s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.090373093 (266h4m25.961456128s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.092053242 (266h4m25.963136287s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.095858979 (266h4m25.966942024s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.096244182 (266h4m25.967327227s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.099898400 (266h4m25.970981445s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:27.793861758 (266h4m27.664944803s) [upload=233226]: Next returned nil segment Dec 16 17:33:27.793883430 (266h4m27.664966465s) [upload=233226]: Waiting for error group managing segments... Dec 16 17:33:28.076919891 (266h4m27.948002936s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.080532791 (266h4m27.951615826s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.081529298 (266h4m27.952612333s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.088220410 (266h4m27.959303455s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.102098585 (266h4m27.97318163s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.110557603 (266h4m27.981640648s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.112773352 (266h4m27.983856387s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.168421046 (266h4m28.039504091s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.175508041 (266h4m28.046591076s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.367477684 (266h4m28.23856073s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.388567654 (266h4m28.259650699s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.393640814 (266h4m28.264723859s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466522674 (266h4m28.337605719s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466547552 (266h4m28.337630597s) [seg_pos={0 0}, upload=233226]: Segment reached optimal threshold of 13 pieces. Dec 16 17:33:28.466920051 (266h4m28.338003096s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466932094 (266h4m28.338015129s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.466960198 (266h4m28.338043243s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466972481 (266h4m28.338055526s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467172281 (266h4m28.338255336s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection Dec 16 17:33:28.467182901 (266h4m28.338265946s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467216766 (266h4m28.338299811s) [seg_pos={0 0}, upload=233226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:33:28.467315314 (266h4m28.338398349s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:33:28.467322477 (266h4m28.338405522s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:33:28.799678512 (266h4m28.670761547s) [upload=233226]: Done uploading segments... Dec 16 17:33:28.799699542 (266h4m28.670782587s) [upload=233226]: Upload finished. err: <nil> Duration: 5.675123473s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.31185225s trace: https://jaeger.dev.storj.io/trace/2130d3e94d4b0821 FAIL (2025-12-16 17:55:43 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37318->45.38.24.2:443: read: connection reset by peer Duration: 2.311877538s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X......X......X..............................] duration: 5.852609121s trace: https://jaeger.dev.storj.io/trace/3a8b2eb64e78cdb7 FAIL (2025-12-16 17:55:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 17:33:25.825412431 (266h4m25.696495476s) [upload=233226]: Starting upload Dec 16 17:33:25.825716299 (266h4m25.696799334s) [upload=233226]: Done starting upload Dec 16 17:33:25.825900039 (266h4m25.696983074s) [upload=233226]: Uploading segments... Dec 16 17:33:25.825986874 (266h4m25.697069919s) [upload=233226]: Got next segment. Inline: false Dec 16 17:33:25.826024115 (266h4m25.69710716s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:33:25.826029405 (266h4m25.69711245s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:33:26.081528100 (266h4m25.952611145s) [seg_pos={0 0}, upload=233226]: Begin upload segment... Dec 16 17:33:26.081578937 (266h4m25.952661972s) [seg_pos={0 0}, upload=233226]: Done begin upload segment. Dec 16 17:33:26.081646916 (266h4m25.952729951s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.081689087 (266h4m25.952772122s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.082557751 (266h4m25.953640796s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083162883 (266h4m25.954245928s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083811628 (266h4m25.954894674s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.084268108 (266h4m25.955351153s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086136556 (266h4m25.957219591s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086764421 (266h4m25.957847456s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.087471016 (266h4m25.958554061s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.088841396 (266h4m25.959924442s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.089031668 (266h4m25.960114703s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.090373093 (266h4m25.961456128s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.092053242 (266h4m25.963136287s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.095858979 (266h4m25.966942024s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.096244182 (266h4m25.967327227s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.099898400 (266h4m25.970981445s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:27.793861758 (266h4m27.664944803s) [upload=233226]: Next returned nil segment Dec 16 17:33:27.793883430 (266h4m27.664966465s) [upload=233226]: Waiting for error group managing segments... Dec 16 17:33:28.076919891 (266h4m27.948002936s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.080532791 (266h4m27.951615826s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.081529298 (266h4m27.952612333s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.088220410 (266h4m27.959303455s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.102098585 (266h4m27.97318163s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.110557603 (266h4m27.981640648s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.112773352 (266h4m27.983856387s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.168421046 (266h4m28.039504091s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.175508041 (266h4m28.046591076s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.367477684 (266h4m28.23856073s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.388567654 (266h4m28.259650699s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.393640814 (266h4m28.264723859s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466522674 (266h4m28.337605719s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466547552 (266h4m28.337630597s) [seg_pos={0 0}, upload=233226]: Segment reached optimal threshold of 13 pieces. Dec 16 17:33:28.466920051 (266h4m28.338003096s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466932094 (266h4m28.338015129s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.466960198 (266h4m28.338043243s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466972481 (266h4m28.338055526s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467172281 (266h4m28.338255336s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection Dec 16 17:33:28.467182901 (266h4m28.338265946s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467216766 (266h4m28.338299811s) [seg_pos={0 0}, upload=233226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:33:28.467315314 (266h4m28.338398349s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:33:28.467322477 (266h4m28.338405522s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:33:28.799678512 (266h4m28.670761547s) [upload=233226]: Done uploading segments... Dec 16 17:33:28.799699542 (266h4m28.670782587s) [upload=233226]: Upload finished. err: <nil> Duration: 5.852640561s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [...............XX...................X...X......XX........X..] duration: 3.566325444s trace: https://jaeger.dev.storj.io/trace/42e4dc6b8c254bbe FAIL (2025-12-16 17:55:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.566345183s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X......X......X.............................] duration: 5.706129404s trace: https://jaeger.dev.storj.io/trace/3a8b2eb64e78cdb7 FAIL (2025-12-16 17:54:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 17:33:25.825412431 (266h4m25.696495476s) [upload=233226]: Starting upload Dec 16 17:33:25.825716299 (266h4m25.696799334s) [upload=233226]: Done starting upload Dec 16 17:33:25.825900039 (266h4m25.696983074s) [upload=233226]: Uploading segments... Dec 16 17:33:25.825986874 (266h4m25.697069919s) [upload=233226]: Got next segment. Inline: false Dec 16 17:33:25.826024115 (266h4m25.69710716s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:33:25.826029405 (266h4m25.69711245s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:33:26.081528100 (266h4m25.952611145s) [seg_pos={0 0}, upload=233226]: Begin upload segment... Dec 16 17:33:26.081578937 (266h4m25.952661972s) [seg_pos={0 0}, upload=233226]: Done begin upload segment. Dec 16 17:33:26.081646916 (266h4m25.952729951s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.081689087 (266h4m25.952772122s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.082557751 (266h4m25.953640796s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083162883 (266h4m25.954245928s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083811628 (266h4m25.954894674s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.084268108 (266h4m25.955351153s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086136556 (266h4m25.957219591s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086764421 (266h4m25.957847456s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.087471016 (266h4m25.958554061s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.088841396 (266h4m25.959924442s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.089031668 (266h4m25.960114703s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.090373093 (266h4m25.961456128s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.092053242 (266h4m25.963136287s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.095858979 (266h4m25.966942024s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.096244182 (266h4m25.967327227s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.099898400 (266h4m25.970981445s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:27.793861758 (266h4m27.664944803s) [upload=233226]: Next returned nil segment Dec 16 17:33:27.793883430 (266h4m27.664966465s) [upload=233226]: Waiting for error group managing segments... Dec 16 17:33:28.076919891 (266h4m27.948002936s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.080532791 (266h4m27.951615826s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.081529298 (266h4m27.952612333s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.088220410 (266h4m27.959303455s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.102098585 (266h4m27.97318163s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.110557603 (266h4m27.981640648s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.112773352 (266h4m27.983856387s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.168421046 (266h4m28.039504091s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.175508041 (266h4m28.046591076s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.367477684 (266h4m28.23856073s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.388567654 (266h4m28.259650699s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.393640814 (266h4m28.264723859s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466522674 (266h4m28.337605719s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466547552 (266h4m28.337630597s) [seg_pos={0 0}, upload=233226]: Segment reached optimal threshold of 13 pieces. Dec 16 17:33:28.466920051 (266h4m28.338003096s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466932094 (266h4m28.338015129s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.466960198 (266h4m28.338043243s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466972481 (266h4m28.338055526s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467172281 (266h4m28.338255336s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection Dec 16 17:33:28.467182901 (266h4m28.338265946s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467216766 (266h4m28.338299811s) [seg_pos={0 0}, upload=233226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:33:28.467315314 (266h4m28.338398349s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:33:28.467322477 (266h4m28.338405522s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:33:28.799678512 (266h4m28.670761547s) [upload=233226]: Done uploading segments... Dec 16 17:33:28.799699542 (266h4m28.670782587s) [upload=233226]: Upload finished. err: <nil> Duration: 5.706156325s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.286325646s trace: https://jaeger.dev.storj.io/trace/2130d3e94d4b0821 FAIL (2025-12-16 17:54:33 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37318->45.38.24.2:443: read: connection reset by peer Duration: 2.286351244s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [................XX...................X...X......XX........X.] duration: 3.523615205s trace: https://jaeger.dev.storj.io/trace/42e4dc6b8c254bbe FAIL (2025-12-16 17:54:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.523645272s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X......X......X............................] duration: 5.740988782s trace: https://jaeger.dev.storj.io/trace/3a8b2eb64e78cdb7 FAIL (2025-12-16 17:53:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 17:33:25.825412431 (266h4m25.696495476s) [upload=233226]: Starting upload Dec 16 17:33:25.825716299 (266h4m25.696799334s) [upload=233226]: Done starting upload Dec 16 17:33:25.825900039 (266h4m25.696983074s) [upload=233226]: Uploading segments... Dec 16 17:33:25.825986874 (266h4m25.697069919s) [upload=233226]: Got next segment. Inline: false Dec 16 17:33:25.826024115 (266h4m25.69710716s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:33:25.826029405 (266h4m25.69711245s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:33:26.081528100 (266h4m25.952611145s) [seg_pos={0 0}, upload=233226]: Begin upload segment... Dec 16 17:33:26.081578937 (266h4m25.952661972s) [seg_pos={0 0}, upload=233226]: Done begin upload segment. Dec 16 17:33:26.081646916 (266h4m25.952729951s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.081689087 (266h4m25.952772122s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.082557751 (266h4m25.953640796s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083162883 (266h4m25.954245928s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083811628 (266h4m25.954894674s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.084268108 (266h4m25.955351153s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086136556 (266h4m25.957219591s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086764421 (266h4m25.957847456s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.087471016 (266h4m25.958554061s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.088841396 (266h4m25.959924442s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.089031668 (266h4m25.960114703s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.090373093 (266h4m25.961456128s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.092053242 (266h4m25.963136287s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.095858979 (266h4m25.966942024s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.096244182 (266h4m25.967327227s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.099898400 (266h4m25.970981445s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:27.793861758 (266h4m27.664944803s) [upload=233226]: Next returned nil segment Dec 16 17:33:27.793883430 (266h4m27.664966465s) [upload=233226]: Waiting for error group managing segments... Dec 16 17:33:28.076919891 (266h4m27.948002936s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.080532791 (266h4m27.951615826s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.081529298 (266h4m27.952612333s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.088220410 (266h4m27.959303455s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.102098585 (266h4m27.97318163s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.110557603 (266h4m27.981640648s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.112773352 (266h4m27.983856387s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.168421046 (266h4m28.039504091s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.175508041 (266h4m28.046591076s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.367477684 (266h4m28.23856073s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.388567654 (266h4m28.259650699s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.393640814 (266h4m28.264723859s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466522674 (266h4m28.337605719s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466547552 (266h4m28.337630597s) [seg_pos={0 0}, upload=233226]: Segment reached optimal threshold of 13 pieces. Dec 16 17:33:28.466920051 (266h4m28.338003096s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466932094 (266h4m28.338015129s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.466960198 (266h4m28.338043243s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466972481 (266h4m28.338055526s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467172281 (266h4m28.338255336s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection Dec 16 17:33:28.467182901 (266h4m28.338265946s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467216766 (266h4m28.338299811s) [seg_pos={0 0}, upload=233226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:33:28.467315314 (266h4m28.338398349s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:33:28.467322477 (266h4m28.338405522s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:33:28.799678512 (266h4m28.670761547s) [upload=233226]: Done uploading segments... Dec 16 17:33:28.799699542 (266h4m28.670782587s) [upload=233226]: Upload finished. err: <nil> Duration: 5.74101347s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.383169834s trace: https://jaeger.dev.storj.io/trace/2130d3e94d4b0821 FAIL (2025-12-16 17:53:20 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37318->45.38.24.2:443: read: connection reset by peer Duration: 2.383197557s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.................XX...................X...X......XX........X] duration: 5.001059846s trace: https://jaeger.dev.storj.io/trace/42e4dc6b8c254bbe FAIL (2025-12-16 17:53:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001099951s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X......X......X...........................] duration: 5.666339932s trace: https://jaeger.dev.storj.io/trace/3a8b2eb64e78cdb7 FAIL (2025-12-16 17:52:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 17:33:25.825412431 (266h4m25.696495476s) [upload=233226]: Starting upload Dec 16 17:33:25.825716299 (266h4m25.696799334s) [upload=233226]: Done starting upload Dec 16 17:33:25.825900039 (266h4m25.696983074s) [upload=233226]: Uploading segments... Dec 16 17:33:25.825986874 (266h4m25.697069919s) [upload=233226]: Got next segment. Inline: false Dec 16 17:33:25.826024115 (266h4m25.69710716s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:33:25.826029405 (266h4m25.69711245s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:33:26.081528100 (266h4m25.952611145s) [seg_pos={0 0}, upload=233226]: Begin upload segment... Dec 16 17:33:26.081578937 (266h4m25.952661972s) [seg_pos={0 0}, upload=233226]: Done begin upload segment. Dec 16 17:33:26.081646916 (266h4m25.952729951s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.081689087 (266h4m25.952772122s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.082557751 (266h4m25.953640796s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083162883 (266h4m25.954245928s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083811628 (266h4m25.954894674s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.084268108 (266h4m25.955351153s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086136556 (266h4m25.957219591s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086764421 (266h4m25.957847456s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.087471016 (266h4m25.958554061s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.088841396 (266h4m25.959924442s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.089031668 (266h4m25.960114703s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.090373093 (266h4m25.961456128s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.092053242 (266h4m25.963136287s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.095858979 (266h4m25.966942024s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.096244182 (266h4m25.967327227s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.099898400 (266h4m25.970981445s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:27.793861758 (266h4m27.664944803s) [upload=233226]: Next returned nil segment Dec 16 17:33:27.793883430 (266h4m27.664966465s) [upload=233226]: Waiting for error group managing segments... Dec 16 17:33:28.076919891 (266h4m27.948002936s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.080532791 (266h4m27.951615826s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.081529298 (266h4m27.952612333s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.088220410 (266h4m27.959303455s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.102098585 (266h4m27.97318163s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.110557603 (266h4m27.981640648s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.112773352 (266h4m27.983856387s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.168421046 (266h4m28.039504091s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.175508041 (266h4m28.046591076s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.367477684 (266h4m28.23856073s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.388567654 (266h4m28.259650699s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.393640814 (266h4m28.264723859s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466522674 (266h4m28.337605719s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466547552 (266h4m28.337630597s) [seg_pos={0 0}, upload=233226]: Segment reached optimal threshold of 13 pieces. Dec 16 17:33:28.466920051 (266h4m28.338003096s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466932094 (266h4m28.338015129s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.466960198 (266h4m28.338043243s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466972481 (266h4m28.338055526s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467172281 (266h4m28.338255336s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection Dec 16 17:33:28.467182901 (266h4m28.338265946s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467216766 (266h4m28.338299811s) [seg_pos={0 0}, upload=233226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:33:28.467315314 (266h4m28.338398349s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:33:28.467322477 (266h4m28.338405522s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:33:28.799678512 (266h4m28.670761547s) [upload=233226]: Done uploading segments... Dec 16 17:33:28.799699542 (266h4m28.670782587s) [upload=233226]: Upload finished. err: <nil> Duration: 5.666382904s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.339878905s trace: https://jaeger.dev.storj.io/trace/2130d3e94d4b0821 FAIL (2025-12-16 17:52:28 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37318->45.38.24.2:443: read: connection reset by peer Duration: 2.339907871s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X......X......X..........................] duration: 5.751542001s trace: https://jaeger.dev.storj.io/trace/3a8b2eb64e78cdb7 FAIL (2025-12-16 17:52:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 17:33:25.825412431 (266h4m25.696495476s) [upload=233226]: Starting upload Dec 16 17:33:25.825716299 (266h4m25.696799334s) [upload=233226]: Done starting upload Dec 16 17:33:25.825900039 (266h4m25.696983074s) [upload=233226]: Uploading segments... Dec 16 17:33:25.825986874 (266h4m25.697069919s) [upload=233226]: Got next segment. Inline: false Dec 16 17:33:25.826024115 (266h4m25.69710716s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:33:25.826029405 (266h4m25.69711245s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:33:26.081528100 (266h4m25.952611145s) [seg_pos={0 0}, upload=233226]: Begin upload segment... Dec 16 17:33:26.081578937 (266h4m25.952661972s) [seg_pos={0 0}, upload=233226]: Done begin upload segment. Dec 16 17:33:26.081646916 (266h4m25.952729951s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.081689087 (266h4m25.952772122s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.082557751 (266h4m25.953640796s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083162883 (266h4m25.954245928s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083811628 (266h4m25.954894674s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.084268108 (266h4m25.955351153s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086136556 (266h4m25.957219591s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086764421 (266h4m25.957847456s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.087471016 (266h4m25.958554061s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.088841396 (266h4m25.959924442s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.089031668 (266h4m25.960114703s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.090373093 (266h4m25.961456128s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.092053242 (266h4m25.963136287s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.095858979 (266h4m25.966942024s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.096244182 (266h4m25.967327227s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.099898400 (266h4m25.970981445s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:27.793861758 (266h4m27.664944803s) [upload=233226]: Next returned nil segment Dec 16 17:33:27.793883430 (266h4m27.664966465s) [upload=233226]: Waiting for error group managing segments... Dec 16 17:33:28.076919891 (266h4m27.948002936s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.080532791 (266h4m27.951615826s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.081529298 (266h4m27.952612333s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.088220410 (266h4m27.959303455s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.102098585 (266h4m27.97318163s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.110557603 (266h4m27.981640648s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.112773352 (266h4m27.983856387s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.168421046 (266h4m28.039504091s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.175508041 (266h4m28.046591076s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.367477684 (266h4m28.23856073s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.388567654 (266h4m28.259650699s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.393640814 (266h4m28.264723859s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466522674 (266h4m28.337605719s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466547552 (266h4m28.337630597s) [seg_pos={0 0}, upload=233226]: Segment reached optimal threshold of 13 pieces. Dec 16 17:33:28.466920051 (266h4m28.338003096s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466932094 (266h4m28.338015129s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.466960198 (266h4m28.338043243s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466972481 (266h4m28.338055526s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467172281 (266h4m28.338255336s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection Dec 16 17:33:28.467182901 (266h4m28.338265946s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467216766 (266h4m28.338299811s) [seg_pos={0 0}, upload=233226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:33:28.467315314 (266h4m28.338398349s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:33:28.467322477 (266h4m28.338405522s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:33:28.799678512 (266h4m28.670761547s) [upload=233226]: Done uploading segments... Dec 16 17:33:28.799699542 (266h4m28.670782587s) [upload=233226]: Upload finished. err: <nil> Duration: 5.751573781s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..................XX...................X...X......XX........] duration: 3.624314109s trace: https://jaeger.dev.storj.io/trace/180e6285647a927 FAIL (2025-12-16 17:52:03 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.624341993s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 4.027393965s trace: https://jaeger.dev.storj.io/trace/2130d3e94d4b0821 FAIL (2025-12-16 17:51:57 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37318->45.38.24.2:443: read: connection reset by peer Duration: 4.027422289s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...................XX...................X...X......XX.......] duration: 3.546440876s trace: https://jaeger.dev.storj.io/trace/180e6285647a927 FAIL (2025-12-16 17:51:30 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.546469148s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.385496241s trace: https://jaeger.dev.storj.io/trace/2130d3e94d4b0821 FAIL (2025-12-16 17:51:19 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37318->45.38.24.2:443: read: connection reset by peer Duration: 2.385525607s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X......X......X.........................] duration: 5.968186811s trace: https://jaeger.dev.storj.io/trace/3a8b2eb64e78cdb7 FAIL (2025-12-16 17:51:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 17:33:25.825412431 (266h4m25.696495476s) [upload=233226]: Starting upload Dec 16 17:33:25.825716299 (266h4m25.696799334s) [upload=233226]: Done starting upload Dec 16 17:33:25.825900039 (266h4m25.696983074s) [upload=233226]: Uploading segments... Dec 16 17:33:25.825986874 (266h4m25.697069919s) [upload=233226]: Got next segment. Inline: false Dec 16 17:33:25.826024115 (266h4m25.69710716s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:33:25.826029405 (266h4m25.69711245s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:33:26.081528100 (266h4m25.952611145s) [seg_pos={0 0}, upload=233226]: Begin upload segment... Dec 16 17:33:26.081578937 (266h4m25.952661972s) [seg_pos={0 0}, upload=233226]: Done begin upload segment. Dec 16 17:33:26.081646916 (266h4m25.952729951s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.081689087 (266h4m25.952772122s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.082557751 (266h4m25.953640796s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083162883 (266h4m25.954245928s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083811628 (266h4m25.954894674s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.084268108 (266h4m25.955351153s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086136556 (266h4m25.957219591s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086764421 (266h4m25.957847456s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.087471016 (266h4m25.958554061s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.088841396 (266h4m25.959924442s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.089031668 (266h4m25.960114703s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.090373093 (266h4m25.961456128s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.092053242 (266h4m25.963136287s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.095858979 (266h4m25.966942024s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.096244182 (266h4m25.967327227s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.099898400 (266h4m25.970981445s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:27.793861758 (266h4m27.664944803s) [upload=233226]: Next returned nil segment Dec 16 17:33:27.793883430 (266h4m27.664966465s) [upload=233226]: Waiting for error group managing segments... Dec 16 17:33:28.076919891 (266h4m27.948002936s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.080532791 (266h4m27.951615826s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.081529298 (266h4m27.952612333s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.088220410 (266h4m27.959303455s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.102098585 (266h4m27.97318163s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.110557603 (266h4m27.981640648s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.112773352 (266h4m27.983856387s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.168421046 (266h4m28.039504091s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.175508041 (266h4m28.046591076s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.367477684 (266h4m28.23856073s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.388567654 (266h4m28.259650699s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.393640814 (266h4m28.264723859s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466522674 (266h4m28.337605719s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466547552 (266h4m28.337630597s) [seg_pos={0 0}, upload=233226]: Segment reached optimal threshold of 13 pieces. Dec 16 17:33:28.466920051 (266h4m28.338003096s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466932094 (266h4m28.338015129s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.466960198 (266h4m28.338043243s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466972481 (266h4m28.338055526s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467172281 (266h4m28.338255336s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection Dec 16 17:33:28.467182901 (266h4m28.338265946s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467216766 (266h4m28.338299811s) [seg_pos={0 0}, upload=233226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:33:28.467315314 (266h4m28.338398349s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:33:28.467322477 (266h4m28.338405522s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:33:28.799678512 (266h4m28.670761547s) [upload=233226]: Done uploading segments... Dec 16 17:33:28.799699542 (266h4m28.670782587s) [upload=233226]: Upload finished. err: <nil> Duration: 5.968220154s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X......X......X........................] duration: 15.175339594s trace: https://jaeger.dev.storj.io/trace/3a8b2eb64e78cdb7 FAIL (2025-12-16 17:50:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 17:33:25.825412431 (266h4m25.696495476s) [upload=233226]: Starting upload Dec 16 17:33:25.825716299 (266h4m25.696799334s) [upload=233226]: Done starting upload Dec 16 17:33:25.825900039 (266h4m25.696983074s) [upload=233226]: Uploading segments... Dec 16 17:33:25.825986874 (266h4m25.697069919s) [upload=233226]: Got next segment. Inline: false Dec 16 17:33:25.826024115 (266h4m25.69710716s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 17:33:25.826029405 (266h4m25.69711245s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 17:33:26.081528100 (266h4m25.952611145s) [seg_pos={0 0}, upload=233226]: Begin upload segment... Dec 16 17:33:26.081578937 (266h4m25.952661972s) [seg_pos={0 0}, upload=233226]: Done begin upload segment. Dec 16 17:33:26.081646916 (266h4m25.952729951s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.081689087 (266h4m25.952772122s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.082557751 (266h4m25.953640796s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083162883 (266h4m25.954245928s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.083811628 (266h4m25.954894674s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.084268108 (266h4m25.955351153s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086136556 (266h4m25.957219591s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.086764421 (266h4m25.957847456s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.087471016 (266h4m25.958554061s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.088841396 (266h4m25.959924442s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.089031668 (266h4m25.960114703s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.090373093 (266h4m25.961456128s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.092053242 (266h4m25.963136287s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.095858979 (266h4m25.966942024s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.096244182 (266h4m25.967327227s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:26.099898400 (266h4m25.970981445s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Uploading piece... Dec 16 17:33:27.793861758 (266h4m27.664944803s) [upload=233226]: Next returned nil segment Dec 16 17:33:27.793883430 (266h4m27.664966465s) [upload=233226]: Waiting for error group managing segments... Dec 16 17:33:28.076919891 (266h4m27.948002936s) [noise=true , address=145.79.227.6:4258 , piece_id=EDRH6NA5BHXMJDBJNFSO527D4RN2TJYMINHBVC33MQHBJ6IB7WAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.080532791 (266h4m27.951615826s) [noise=true , address=145.79.227.45:4231 , piece_id=ID7WWF3KA6AO4ATFB66X6QH4N6WJNMXCFVZKJ7PTOTR3HGIXD5EA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.081529298 (266h4m27.952612333s) [noise=true , address=145.79.227.29:4943 , piece_id=DCSFSVECGDGLPTWDPBRG6XOIRLNMHBDFS5OBRXLTBAFJIHX7RK2A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.088220410 (266h4m27.959303455s) [noise=true , address=145.79.227.34:4906 , piece_id=2XYGUEMMEHM2ASIJPGTFGL4UFXSDT3QWRAYZVU22DC4ILLYWDFZQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.102098585 (266h4m27.97318163s) [noise=true , address=145.79.227.49:4338 , piece_id=DGC6J6UB6FPGBIIRRIFITEOCR5PNIJCHTVA2C7KY73HO3VNVWKIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.110557603 (266h4m27.981640648s) [noise=true , address=145.79.227.41:4086 , piece_id=K3TJX2MVDHLI4OEUXW2KFKRFRRXABJM5HNOXRSD2V6DNNISD2GXA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.112773352 (266h4m27.983856387s) [noise=true , address=145.79.227.51:4410 , piece_id=V7SQP2FRXVJMAOXAZCGMWXPVFQ3Q4HW4SI4AXDJGTNTSCZZF72TQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.168421046 (266h4m28.039504091s) [noise=true , address=145.79.227.46:4602 , piece_id=QGZMOQ6EFBKU56GMNOFF32O7KNJUQVWRZ73H4YDUSJG7F5BU7LAA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.175508041 (266h4m28.046591076s) [noise=true , address=145.79.227.42:4122 , piece_id=OAW7CSLO63IMAUHYDJG6EMJER7JPB5UREMN2ZNYMRZUBKGU7SEWA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.367477684 (266h4m28.23856073s) [noise=true , address=77.90.31.12:4590 , piece_id=IJWM6R2CAELS35NH4ITFHW5ZVWZBRIGFUIWMGXULK223EIURPC7A, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.388567654 (266h4m28.259650699s) [noise=true , address=45.91.116.13:4155 , piece_id=MBX5INRSFOWWZSCCDVOLZZTQFO2UOUHXDVEYZAI4Q3VVNHBB5D2Q, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.393640814 (266h4m28.264723859s) [noise=true , address=45.91.116.6:4974 , piece_id=OA6N55DFJA2W5HVIXK2KNFWA4DMUH4YDAIYR32QXDJAUAJWLJQVQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466522674 (266h4m28.337605719s) [noise=true , address=45.91.116.14:4192 , piece_id=7ZZEJH7EYBYISVPBKSQQV2GSSQY3T6ARXHTDFXDKAYU6JCQG4VRQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: <nil> Dec 16 17:33:28.466547552 (266h4m28.337630597s) [seg_pos={0 0}, upload=233226]: Segment reached optimal threshold of 13 pieces. Dec 16 17:33:28.466920051 (266h4m28.338003096s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466932094 (266h4m28.338015129s) [noise=true , address=77.90.31.16:4736 , piece_id=A6RIKEEKRFSVA2AF2276L3L7D5Y76REPSH3N4PYYFZVTFUR7JPJA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.466960198 (266h4m28.338043243s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 16 17:33:28.466972481 (266h4m28.338055526s) [noise=true , address=45.91.116.10:4082 , piece_id=HUZ5LHJQVZRW2KBEHABVGMUDSSBSLID7CREECI6MWVFK63YAUQIA, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467172281 (266h4m28.338255336s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37906->77.90.31.14:4661: use of closed network connection Dec 16 17:33:28.467182901 (266h4m28.338265946s) [noise=true , address=77.90.31.14:4661 , piece_id=KMGEJGPWC5SV547K5OLK7VGZLANKSR2DI5DSEYPGECPFDBFWUZSQ, seg_pos={0 0}, upload=233226]: Long tail cancellation due to optimal threshold Dec 16 17:33:28.467216766 (266h4m28.338299811s) [seg_pos={0 0}, upload=233226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 17:33:28.467315314 (266h4m28.338398349s) [upload=233226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 17:33:28.467322477 (266h4m28.338405522s) [upload=233226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 17:33:28.799678512 (266h4m28.670761547s) [upload=233226]: Done uploading segments... Dec 16 17:33:28.799699542 (266h4m28.670782587s) [upload=233226]: Upload finished. err: <nil> Duration: 15.175371934s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [X...................XX...................X...X......XX......] duration: 3.602594843s trace: https://jaeger.dev.storj.io/trace/180e6285647a927 FAIL (2025-12-16 17:50:42 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.602617516s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.444621726s trace: https://jaeger.dev.storj.io/trace/2130d3e94d4b0821 FAIL (2025-12-16 17:50:36 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37318->45.38.24.2:443: read: connection reset by peer Duration: 2.44465028s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)