Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.186550715s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 14:00:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 6.18658449s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 7.778672623s trace: https://jaeger.dev.storj.io/trace/129f27707330b16c FAIL (2025-11-16 14:00:20 UTC) orbiter warning: context deadline exceeded Duration: 7.778752886s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.X........X.........XX....X......X..XX....X.................] duration: 5.001465486s trace: https://jaeger.dev.storj.io/trace/69ae5765386935f3 FAIL (2025-11-16 13:59:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001507847s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.729434754s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 13:59:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 6.729467617s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.050681481s trace: https://jaeger.dev.storj.io/trace/129f27707330b16c FAIL (2025-11-16 13:59:28 UTC) orbiter warning: context deadline exceeded Duration: 6.050708091s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.170721191s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 13:59:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 6.170740788s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [..X........X.........XX....X......X..XX....X................] duration: 3.554843864s trace: https://jaeger.dev.storj.io/trace/69ae5765386935f3 FAIL (2025-11-16 13:59:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.554881337s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 8.909001515s trace: https://jaeger.dev.storj.io/trace/129f27707330b16c FAIL (2025-11-16 13:58:34 UTC) orbiter warning: context deadline exceeded Duration: 8.909032684s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 6.303371699s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 13:58:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 6.303397087s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [...X........X.........XX....X......X..XX....X...............] duration: 3.728809724s trace: https://jaeger.dev.storj.io/trace/69ae5765386935f3 FAIL (2025-11-16 13:58:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.728889196s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 13.164445259s trace: https://jaeger.dev.storj.io/trace/129f27707330b16c FAIL (2025-11-16 13:58:16 UTC) orbiter warning: context deadline exceeded Duration: 13.164499162s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 17.023271457s trace: https://jaeger.dev.storj.io/trace/129f27707330b16c FAIL (2025-11-16 13:57:47 UTC) orbiter warning: context deadline exceeded Duration: 17.023297217s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [....X........X.........XX....X......X..XX....X..............] duration: 5.001148506s trace: https://jaeger.dev.storj.io/trace/69ae5765386935f3 FAIL (2025-11-16 13:57:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001166882s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.261158885s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 13:57:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 6.261189673s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [X....X........X.........XX....X......X..XX....X.............] duration: 3.687562919s trace: https://jaeger.dev.storj.io/trace/69ae5765386935f3 FAIL (2025-11-16 13:57:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.687588999s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.636313671s trace: https://jaeger.dev.storj.io/trace/129f27707330b16c FAIL (2025-11-16 13:57:14 UTC) orbiter warning: context deadline exceeded Duration: 6.636342806s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 12.643614374s trace: https://jaeger.dev.storj.io/trace/129f27707330b16c FAIL (2025-11-16 13:56:48 UTC) orbiter warning: context deadline exceeded Duration: 12.643649812s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.071365744s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 13:56:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 6.071469242s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 12.692784436s trace: https://jaeger.dev.storj.io/trace/129f27707330b16c FAIL (2025-11-16 13:56:24 UTC) orbiter warning: context deadline exceeded Duration: 12.692818461s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [XX....X........X.........XX....X......X..XX....X............] duration: 3.451882469s trace: https://jaeger.dev.storj.io/trace/69ae5765386935f3 FAIL (2025-11-16 13:56:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.451911694s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 10.965255312s trace: https://jaeger.dev.storj.io/trace/129f27707330b16c FAIL (2025-11-16 13:55:54 UTC) orbiter warning: context deadline exceeded Duration: 10.965283876s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.263234125s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 13:55:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 6.263265405s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 11.469539309s trace: https://jaeger.dev.storj.io/trace/129f27707330b16c FAIL (2025-11-16 13:55:32 UTC) orbiter warning: context deadline exceeded Duration: 11.469575438s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.XX....X........X.........XX....X......X..XX....X...........] duration: 3.779810691s trace: https://jaeger.dev.storj.io/trace/69ae5765386935f3 FAIL (2025-11-16 13:55:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.779854425s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 7.983413762s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 13:54:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 7.983460743s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [..XX....X........X.........XX....X......X..XX....X..........] duration: 3.512033481s trace: https://jaeger.dev.storj.io/trace/69ae5765386935f3 FAIL (2025-11-16 13:54:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.512061434s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 7.152701013s trace: https://jaeger.dev.storj.io/trace/129f27707330b16c FAIL (2025-11-16 13:54:21 UTC) orbiter warning: context deadline exceeded Duration: 7.1527301s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.025701157s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 13:53:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 6.025733498s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [...XX....X........X.........XX....X......X..XX....X.........] duration: 3.757507389s trace: https://jaeger.dev.storj.io/trace/69ae5765386935f3 FAIL (2025-11-16 13:53:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.757533349s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.034915304s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 13:53:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 6.034940872s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 15.382468221s trace: https://jaeger.dev.storj.io/trace/129f27707330b16c FAIL (2025-11-16 13:53:29 UTC) orbiter warning: context deadline exceeded Duration: 15.382504622s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [....XX....X........X.........XX....X......X..XX....X........] duration: 5.002401663s trace: https://jaeger.dev.storj.io/trace/69ae5765386935f3 FAIL (2025-11-16 13:53:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002424206s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.....XX....X........X.........XX....X......X..XX....X.......] duration: 4.871484447s trace: https://jaeger.dev.storj.io/trace/69ae5765386935f3 FAIL (2025-11-16 13:53:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.871516628s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 9.948658344s trace: https://jaeger.dev.storj.io/trace/129f27707330b16c FAIL (2025-11-16 13:53:02 UTC) orbiter warning: context deadline exceeded Duration: 9.948715553s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 22.228692831s trace: https://jaeger.dev.storj.io/trace/129f27707330b16c FAIL (2025-11-16 13:52:50 UTC) orbiter warning: context deadline exceeded Duration: 22.22874542s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.059354123s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 13:52:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 6.059381044s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [......XX....X........X.........XX....X......X..XX....X......] duration: 3.625142956s trace: https://jaeger.dev.storj.io/trace/69ae5765386935f3 FAIL (2025-11-16 13:52:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.625245971s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.......XX....X........X.........XX....X......X..XX....X.....] duration: 3.511881649s trace: https://jaeger.dev.storj.io/trace/69ae5765386935f3 FAIL (2025-11-16 13:51:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.511951483s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.021246313s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 13:51:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 6.021303513s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 15.207180085s trace: https://jaeger.dev.storj.io/trace/129f27707330b16c FAIL (2025-11-16 13:51:31 UTC) orbiter warning: context deadline exceeded Duration: 15.207209761s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [........XX....X........X.........XX....X......X..XX....X....] duration: 3.59786856s trace: https://jaeger.dev.storj.io/trace/69ae5765386935f3 FAIL (2025-11-16 13:51:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.597884381s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.........XX....X........X.........XX....X......X..XX....X...] duration: 3.465168197s trace: https://jaeger.dev.storj.io/trace/69ae5765386935f3 FAIL (2025-11-16 13:51:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.465208574s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.013530523s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 13:51:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 6.013563366s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 17.879044483s trace: https://jaeger.dev.storj.io/trace/129f27707330b16c FAIL (2025-11-16 13:50:23 UTC) orbiter warning: context deadline exceeded Duration: 17.87907953s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.346485116s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 13:50:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 6.346510584s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [..........XX....X........X.........XX....X......X..XX....X..] duration: 5.002068707s trace: https://jaeger.dev.storj.io/trace/69ae5765386935f3 FAIL (2025-11-16 13:49:59 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002094236s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.08105854s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 13:49:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 6.0810852s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [...........XX....X........X.........XX....X......X..XX....X.] duration: 4.915768052s trace: https://jaeger.dev.storj.io/trace/69ae5765386935f3 FAIL (2025-11-16 13:49:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.915876069s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 14.919930781s trace: https://jaeger.dev.storj.io/trace/129f27707330b16c FAIL (2025-11-16 13:49:32 UTC) orbiter warning: context deadline exceeded Duration: 14.919956221s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [............XX....X........X.........XX....X......X..XX....X] duration: 5.002044655s trace: https://jaeger.dev.storj.io/trace/69ae5765386935f3 FAIL (2025-11-16 13:48:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002068559s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)