Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.914310651s trace: https://jaeger.dev.storj.io/trace/640930c436a3dd40 FAIL (2025-11-15 22:41:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA) context deadline exceeded; piecestore: (Node ID: 12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, Piece ID: AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA) context deadline exceeded; piecestore: (Node ID: 1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, Piece ID: TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA) context deadline exceeded; piecestore: (Node ID: 12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, Piece ID: DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A) context deadline exceeded ======LOGS====== Nov 15 22:26:23.884265358 (158h22m22.43573582s) [upload=139438]: Got next segment. Inline: false Nov 15 22:26:23.884310675 (158h22m22.435781137s) [upload=139438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 22:26:23.884317198 (158h22m22.435787649s) [upload=139438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 22:26:24.145294674 (158h22m22.696765135s) [seg_pos={0 0}, upload=139438]: Begin upload segment... Nov 15 22:26:24.145353216 (158h22m22.696823667s) [seg_pos={0 0}, upload=139438]: Done begin upload segment. Nov 15 22:26:24.145389284 (158h22m22.696859745s) [noise=true , address=145.79.227.28:4727 , piece_id=JTVEZRINCPKC7U532YI62SEXI25AXEE6ZBXEXWGTI4IWVEPTVWSQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.145586640 (158h22m22.697057091s) [noise=true , address=145.79.227.21:4473 , piece_id=5WUFKUK7GAWMRPMKCGJD3QAXWLE2AH4SP2WN653JYGWZX5OWWGBA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146324976 (158h22m22.697795437s) [noise=true , address=145.79.227.11:4731 , piece_id=2BUZ5Q4KA4MG6EBYHXRICRVAE6MBL6QXMJSIZFV66RBCJ7L6MDKQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146351477 (158h22m22.697821938s) [noise=true , address=45.91.116.11:4117 , piece_id=SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146783870 (158h22m22.698254331s) [noise=true , address=145.79.227.18:4005 , piece_id=AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146833304 (158h22m22.698303765s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147216514 (158h22m22.698686975s) [noise=true , address=145.79.227.10:4402 , piece_id=HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147263273 (158h22m22.698733734s) [noise=true , address=45.91.116.14:4190 , piece_id=X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147675318 (158h22m22.699145799s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147754558 (158h22m22.699225019s) [noise=true , address=145.79.227.20:4439 , piece_id=TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148345744 (158h22m22.699816205s) [noise=true , address=45.91.116.10:4083 , piece_id=A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148407592 (158h22m22.699878053s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148798777 (158h22m22.700269228s) [noise=true , address=145.79.227.31:4799 , piece_id=LT3JYVZJ2A3I4VPLCEYONSNIPRBB4E2XSODHWL572IB3NFCM26JA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148858250 (158h22m22.700328711s) [noise=true , address=145.79.227.27:4691 , piece_id=AN2YAQFX4APSPXZ37L3THXZDLOU3NZJXZX3GFPQ6GOP2XYMYHJMA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.149166006 (158h22m22.700636467s) [noise=true , address=145.79.227.8:4329 , piece_id=C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.149373912 (158h22m22.700844373s) [noise=true , address=77.90.31.12:4592 , piece_id=DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:25.877223267 (158h22m24.428693728s) [upload=139438]: Next returned nil segment Nov 15 22:26:25.877245679 (158h22m24.42871614s) [upload=139438]: Waiting for error group managing segments... Nov 15 22:26:26.154633386 (158h22m24.706103847s) [noise=true , address=145.79.227.31:4799 , piece_id=LT3JYVZJ2A3I4VPLCEYONSNIPRBB4E2XSODHWL572IB3NFCM26JA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.172371583 (158h22m24.723842044s) [noise=true , address=145.79.227.28:4727 , piece_id=JTVEZRINCPKC7U532YI62SEXI25AXEE6ZBXEXWGTI4IWVEPTVWSQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.173052429 (158h22m24.72452289s) [noise=true , address=145.79.227.20:4439 , piece_id=TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.174665170 (158h22m24.726135621s) [noise=true , address=145.79.227.10:4402 , piece_id=HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.175717564 (158h22m24.727188015s) [noise=true , address=145.79.227.11:4731 , piece_id=2BUZ5Q4KA4MG6EBYHXRICRVAE6MBL6QXMJSIZFV66RBCJ7L6MDKQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.203023491 (158h22m24.754493952s) [noise=true , address=45.91.116.14:4190 , piece_id=X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.211675806 (158h22m24.763146267s) [noise=true , address=145.79.227.21:4473 , piece_id=5WUFKUK7GAWMRPMKCGJD3QAXWLE2AH4SP2WN653JYGWZX5OWWGBA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.241230605 (158h22m24.792701066s) [noise=true , address=145.79.227.18:4005 , piece_id=AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.265904560 (158h22m24.817375011s) [noise=true , address=45.91.116.11:4117 , piece_id=SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.267751306 (158h22m24.819221757s) [noise=true , address=145.79.227.27:4691 , piece_id=AN2YAQFX4APSPXZ37L3THXZDLOU3NZJXZX3GFPQ6GOP2XYMYHJMA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.297086917 (158h22m24.848557379s) [noise=true , address=45.91.116.10:4083 , piece_id=A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.396803830 (158h22m24.948274281s) [noise=true , address=77.90.31.12:4592 , piece_id=DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:51.578031970 (158h22m50.129502431s) [noise=true , address=145.79.227.8:4329 , piece_id=C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:51.578058921 (158h22m50.129529382s) [seg_pos={0 0}, upload=139438]: Segment reached optimal threshold of 13 pieces. Nov 15 22:26:51.578335898 (158h22m50.129806369s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Nov 15 22:26:51.578366757 (158h22m50.129837218s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.578860117 (158h22m50.130330588s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection Nov 15 22:26:51.578892328 (158h22m50.130362789s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.579086789 (158h22m50.13055726s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection Nov 15 22:26:51.579099853 (158h22m50.130570314s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.579150771 (158h22m50.130621232s) [seg_pos={0 0}, upload=139438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 22:26:51.579271630 (158h22m50.130742101s) [upload=139438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 22:26:51.579279916 (158h22m50.130750377s) [upload=139438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 22:26:51.887479665 (158h22m50.438950126s) [upload=139438]: Done uploading segments... Nov 15 22:26:51.887500194 (158h22m50.438970655s) [upload=139438]: Upload finished. err: <nil> Duration: 5.914340979s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 6.057359107s trace: https://jaeger.dev.storj.io/trace/640930c436a3dd40 FAIL (2025-11-15 22:40:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA) context deadline exceeded; piecestore: (Node ID: 12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, Piece ID: AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA) context deadline exceeded; piecestore: (Node ID: 1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, Piece ID: TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA) context deadline exceeded; piecestore: (Node ID: 12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, Piece ID: DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A) context deadline exceeded ======LOGS====== Nov 15 22:26:23.884265358 (158h22m22.43573582s) [upload=139438]: Got next segment. Inline: false Nov 15 22:26:23.884310675 (158h22m22.435781137s) [upload=139438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 22:26:23.884317198 (158h22m22.435787649s) [upload=139438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 22:26:24.145294674 (158h22m22.696765135s) [seg_pos={0 0}, upload=139438]: Begin upload segment... Nov 15 22:26:24.145353216 (158h22m22.696823667s) [seg_pos={0 0}, upload=139438]: Done begin upload segment. Nov 15 22:26:24.145389284 (158h22m22.696859745s) [noise=true , address=145.79.227.28:4727 , piece_id=JTVEZRINCPKC7U532YI62SEXI25AXEE6ZBXEXWGTI4IWVEPTVWSQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.145586640 (158h22m22.697057091s) [noise=true , address=145.79.227.21:4473 , piece_id=5WUFKUK7GAWMRPMKCGJD3QAXWLE2AH4SP2WN653JYGWZX5OWWGBA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146324976 (158h22m22.697795437s) [noise=true , address=145.79.227.11:4731 , piece_id=2BUZ5Q4KA4MG6EBYHXRICRVAE6MBL6QXMJSIZFV66RBCJ7L6MDKQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146351477 (158h22m22.697821938s) [noise=true , address=45.91.116.11:4117 , piece_id=SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146783870 (158h22m22.698254331s) [noise=true , address=145.79.227.18:4005 , piece_id=AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146833304 (158h22m22.698303765s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147216514 (158h22m22.698686975s) [noise=true , address=145.79.227.10:4402 , piece_id=HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147263273 (158h22m22.698733734s) [noise=true , address=45.91.116.14:4190 , piece_id=X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147675318 (158h22m22.699145799s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147754558 (158h22m22.699225019s) [noise=true , address=145.79.227.20:4439 , piece_id=TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148345744 (158h22m22.699816205s) [noise=true , address=45.91.116.10:4083 , piece_id=A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148407592 (158h22m22.699878053s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148798777 (158h22m22.700269228s) [noise=true , address=145.79.227.31:4799 , piece_id=LT3JYVZJ2A3I4VPLCEYONSNIPRBB4E2XSODHWL572IB3NFCM26JA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148858250 (158h22m22.700328711s) [noise=true , address=145.79.227.27:4691 , piece_id=AN2YAQFX4APSPXZ37L3THXZDLOU3NZJXZX3GFPQ6GOP2XYMYHJMA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.149166006 (158h22m22.700636467s) [noise=true , address=145.79.227.8:4329 , piece_id=C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.149373912 (158h22m22.700844373s) [noise=true , address=77.90.31.12:4592 , piece_id=DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:25.877223267 (158h22m24.428693728s) [upload=139438]: Next returned nil segment Nov 15 22:26:25.877245679 (158h22m24.42871614s) [upload=139438]: Waiting for error group managing segments... Nov 15 22:26:26.154633386 (158h22m24.706103847s) [noise=true , address=145.79.227.31:4799 , piece_id=LT3JYVZJ2A3I4VPLCEYONSNIPRBB4E2XSODHWL572IB3NFCM26JA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.172371583 (158h22m24.723842044s) [noise=true , address=145.79.227.28:4727 , piece_id=JTVEZRINCPKC7U532YI62SEXI25AXEE6ZBXEXWGTI4IWVEPTVWSQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.173052429 (158h22m24.72452289s) [noise=true , address=145.79.227.20:4439 , piece_id=TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.174665170 (158h22m24.726135621s) [noise=true , address=145.79.227.10:4402 , piece_id=HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.175717564 (158h22m24.727188015s) [noise=true , address=145.79.227.11:4731 , piece_id=2BUZ5Q4KA4MG6EBYHXRICRVAE6MBL6QXMJSIZFV66RBCJ7L6MDKQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.203023491 (158h22m24.754493952s) [noise=true , address=45.91.116.14:4190 , piece_id=X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.211675806 (158h22m24.763146267s) [noise=true , address=145.79.227.21:4473 , piece_id=5WUFKUK7GAWMRPMKCGJD3QAXWLE2AH4SP2WN653JYGWZX5OWWGBA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.241230605 (158h22m24.792701066s) [noise=true , address=145.79.227.18:4005 , piece_id=AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.265904560 (158h22m24.817375011s) [noise=true , address=45.91.116.11:4117 , piece_id=SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.267751306 (158h22m24.819221757s) [noise=true , address=145.79.227.27:4691 , piece_id=AN2YAQFX4APSPXZ37L3THXZDLOU3NZJXZX3GFPQ6GOP2XYMYHJMA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.297086917 (158h22m24.848557379s) [noise=true , address=45.91.116.10:4083 , piece_id=A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.396803830 (158h22m24.948274281s) [noise=true , address=77.90.31.12:4592 , piece_id=DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:51.578031970 (158h22m50.129502431s) [noise=true , address=145.79.227.8:4329 , piece_id=C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:51.578058921 (158h22m50.129529382s) [seg_pos={0 0}, upload=139438]: Segment reached optimal threshold of 13 pieces. Nov 15 22:26:51.578335898 (158h22m50.129806369s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Nov 15 22:26:51.578366757 (158h22m50.129837218s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.578860117 (158h22m50.130330588s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection Nov 15 22:26:51.578892328 (158h22m50.130362789s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.579086789 (158h22m50.13055726s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection Nov 15 22:26:51.579099853 (158h22m50.130570314s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.579150771 (158h22m50.130621232s) [seg_pos={0 0}, upload=139438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 22:26:51.579271630 (158h22m50.130742101s) [upload=139438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 22:26:51.579279916 (158h22m50.130750377s) [upload=139438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 22:26:51.887479665 (158h22m50.438950126s) [upload=139438]: Done uploading segments... Nov 15 22:26:51.887500194 (158h22m50.438970655s) [upload=139438]: Upload finished. err: <nil> Duration: 6.057389585s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.905033559s trace: https://jaeger.dev.storj.io/trace/640930c436a3dd40 FAIL (2025-11-15 22:40:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA) context deadline exceeded; piecestore: (Node ID: 12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, Piece ID: AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA) context deadline exceeded; piecestore: (Node ID: 1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, Piece ID: TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA) context deadline exceeded; piecestore: (Node ID: 12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, Piece ID: DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A) context deadline exceeded ======LOGS====== Nov 15 22:26:23.884265358 (158h22m22.43573582s) [upload=139438]: Got next segment. Inline: false Nov 15 22:26:23.884310675 (158h22m22.435781137s) [upload=139438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 22:26:23.884317198 (158h22m22.435787649s) [upload=139438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 22:26:24.145294674 (158h22m22.696765135s) [seg_pos={0 0}, upload=139438]: Begin upload segment... Nov 15 22:26:24.145353216 (158h22m22.696823667s) [seg_pos={0 0}, upload=139438]: Done begin upload segment. Nov 15 22:26:24.145389284 (158h22m22.696859745s) [noise=true , address=145.79.227.28:4727 , piece_id=JTVEZRINCPKC7U532YI62SEXI25AXEE6ZBXEXWGTI4IWVEPTVWSQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.145586640 (158h22m22.697057091s) [noise=true , address=145.79.227.21:4473 , piece_id=5WUFKUK7GAWMRPMKCGJD3QAXWLE2AH4SP2WN653JYGWZX5OWWGBA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146324976 (158h22m22.697795437s) [noise=true , address=145.79.227.11:4731 , piece_id=2BUZ5Q4KA4MG6EBYHXRICRVAE6MBL6QXMJSIZFV66RBCJ7L6MDKQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146351477 (158h22m22.697821938s) [noise=true , address=45.91.116.11:4117 , piece_id=SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146783870 (158h22m22.698254331s) [noise=true , address=145.79.227.18:4005 , piece_id=AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146833304 (158h22m22.698303765s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147216514 (158h22m22.698686975s) [noise=true , address=145.79.227.10:4402 , piece_id=HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147263273 (158h22m22.698733734s) [noise=true , address=45.91.116.14:4190 , piece_id=X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147675318 (158h22m22.699145799s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147754558 (158h22m22.699225019s) [noise=true , address=145.79.227.20:4439 , piece_id=TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148345744 (158h22m22.699816205s) [noise=true , address=45.91.116.10:4083 , piece_id=A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148407592 (158h22m22.699878053s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148798777 (158h22m22.700269228s) [noise=true , address=145.79.227.31:4799 , piece_id=LT3JYVZJ2A3I4VPLCEYONSNIPRBB4E2XSODHWL572IB3NFCM26JA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148858250 (158h22m22.700328711s) [noise=true , address=145.79.227.27:4691 , piece_id=AN2YAQFX4APSPXZ37L3THXZDLOU3NZJXZX3GFPQ6GOP2XYMYHJMA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.149166006 (158h22m22.700636467s) [noise=true , address=145.79.227.8:4329 , piece_id=C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.149373912 (158h22m22.700844373s) [noise=true , address=77.90.31.12:4592 , piece_id=DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:25.877223267 (158h22m24.428693728s) [upload=139438]: Next returned nil segment Nov 15 22:26:25.877245679 (158h22m24.42871614s) [upload=139438]: Waiting for error group managing segments... Nov 15 22:26:26.154633386 (158h22m24.706103847s) [noise=true , address=145.79.227.31:4799 , piece_id=LT3JYVZJ2A3I4VPLCEYONSNIPRBB4E2XSODHWL572IB3NFCM26JA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.172371583 (158h22m24.723842044s) [noise=true , address=145.79.227.28:4727 , piece_id=JTVEZRINCPKC7U532YI62SEXI25AXEE6ZBXEXWGTI4IWVEPTVWSQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.173052429 (158h22m24.72452289s) [noise=true , address=145.79.227.20:4439 , piece_id=TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.174665170 (158h22m24.726135621s) [noise=true , address=145.79.227.10:4402 , piece_id=HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.175717564 (158h22m24.727188015s) [noise=true , address=145.79.227.11:4731 , piece_id=2BUZ5Q4KA4MG6EBYHXRICRVAE6MBL6QXMJSIZFV66RBCJ7L6MDKQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.203023491 (158h22m24.754493952s) [noise=true , address=45.91.116.14:4190 , piece_id=X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.211675806 (158h22m24.763146267s) [noise=true , address=145.79.227.21:4473 , piece_id=5WUFKUK7GAWMRPMKCGJD3QAXWLE2AH4SP2WN653JYGWZX5OWWGBA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.241230605 (158h22m24.792701066s) [noise=true , address=145.79.227.18:4005 , piece_id=AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.265904560 (158h22m24.817375011s) [noise=true , address=45.91.116.11:4117 , piece_id=SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.267751306 (158h22m24.819221757s) [noise=true , address=145.79.227.27:4691 , piece_id=AN2YAQFX4APSPXZ37L3THXZDLOU3NZJXZX3GFPQ6GOP2XYMYHJMA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.297086917 (158h22m24.848557379s) [noise=true , address=45.91.116.10:4083 , piece_id=A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.396803830 (158h22m24.948274281s) [noise=true , address=77.90.31.12:4592 , piece_id=DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:51.578031970 (158h22m50.129502431s) [noise=true , address=145.79.227.8:4329 , piece_id=C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:51.578058921 (158h22m50.129529382s) [seg_pos={0 0}, upload=139438]: Segment reached optimal threshold of 13 pieces. Nov 15 22:26:51.578335898 (158h22m50.129806369s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Nov 15 22:26:51.578366757 (158h22m50.129837218s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.578860117 (158h22m50.130330588s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection Nov 15 22:26:51.578892328 (158h22m50.130362789s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.579086789 (158h22m50.13055726s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection Nov 15 22:26:51.579099853 (158h22m50.130570314s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.579150771 (158h22m50.130621232s) [seg_pos={0 0}, upload=139438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 22:26:51.579271630 (158h22m50.130742101s) [upload=139438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 22:26:51.579279916 (158h22m50.130750377s) [upload=139438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 22:26:51.887479665 (158h22m50.438950126s) [upload=139438]: Done uploading segments... Nov 15 22:26:51.887500194 (158h22m50.438970655s) [upload=139438]: Upload finished. err: <nil> Duration: 5.905095417s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.949119459s trace: https://jaeger.dev.storj.io/trace/640930c436a3dd40 FAIL (2025-11-15 22:39:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA) context deadline exceeded; piecestore: (Node ID: 12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, Piece ID: AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA) context deadline exceeded; piecestore: (Node ID: 1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, Piece ID: TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA) context deadline exceeded; piecestore: (Node ID: 12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, Piece ID: DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A) context deadline exceeded ======LOGS====== Nov 15 22:26:23.884265358 (158h22m22.43573582s) [upload=139438]: Got next segment. Inline: false Nov 15 22:26:23.884310675 (158h22m22.435781137s) [upload=139438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 22:26:23.884317198 (158h22m22.435787649s) [upload=139438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 22:26:24.145294674 (158h22m22.696765135s) [seg_pos={0 0}, upload=139438]: Begin upload segment... Nov 15 22:26:24.145353216 (158h22m22.696823667s) [seg_pos={0 0}, upload=139438]: Done begin upload segment. Nov 15 22:26:24.145389284 (158h22m22.696859745s) [noise=true , address=145.79.227.28:4727 , piece_id=JTVEZRINCPKC7U532YI62SEXI25AXEE6ZBXEXWGTI4IWVEPTVWSQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.145586640 (158h22m22.697057091s) [noise=true , address=145.79.227.21:4473 , piece_id=5WUFKUK7GAWMRPMKCGJD3QAXWLE2AH4SP2WN653JYGWZX5OWWGBA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146324976 (158h22m22.697795437s) [noise=true , address=145.79.227.11:4731 , piece_id=2BUZ5Q4KA4MG6EBYHXRICRVAE6MBL6QXMJSIZFV66RBCJ7L6MDKQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146351477 (158h22m22.697821938s) [noise=true , address=45.91.116.11:4117 , piece_id=SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146783870 (158h22m22.698254331s) [noise=true , address=145.79.227.18:4005 , piece_id=AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146833304 (158h22m22.698303765s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147216514 (158h22m22.698686975s) [noise=true , address=145.79.227.10:4402 , piece_id=HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147263273 (158h22m22.698733734s) [noise=true , address=45.91.116.14:4190 , piece_id=X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147675318 (158h22m22.699145799s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147754558 (158h22m22.699225019s) [noise=true , address=145.79.227.20:4439 , piece_id=TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148345744 (158h22m22.699816205s) [noise=true , address=45.91.116.10:4083 , piece_id=A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148407592 (158h22m22.699878053s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148798777 (158h22m22.700269228s) [noise=true , address=145.79.227.31:4799 , piece_id=LT3JYVZJ2A3I4VPLCEYONSNIPRBB4E2XSODHWL572IB3NFCM26JA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148858250 (158h22m22.700328711s) [noise=true , address=145.79.227.27:4691 , piece_id=AN2YAQFX4APSPXZ37L3THXZDLOU3NZJXZX3GFPQ6GOP2XYMYHJMA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.149166006 (158h22m22.700636467s) [noise=true , address=145.79.227.8:4329 , piece_id=C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.149373912 (158h22m22.700844373s) [noise=true , address=77.90.31.12:4592 , piece_id=DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:25.877223267 (158h22m24.428693728s) [upload=139438]: Next returned nil segment Nov 15 22:26:25.877245679 (158h22m24.42871614s) [upload=139438]: Waiting for error group managing segments... Nov 15 22:26:26.154633386 (158h22m24.706103847s) [noise=true , address=145.79.227.31:4799 , piece_id=LT3JYVZJ2A3I4VPLCEYONSNIPRBB4E2XSODHWL572IB3NFCM26JA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.172371583 (158h22m24.723842044s) [noise=true , address=145.79.227.28:4727 , piece_id=JTVEZRINCPKC7U532YI62SEXI25AXEE6ZBXEXWGTI4IWVEPTVWSQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.173052429 (158h22m24.72452289s) [noise=true , address=145.79.227.20:4439 , piece_id=TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.174665170 (158h22m24.726135621s) [noise=true , address=145.79.227.10:4402 , piece_id=HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.175717564 (158h22m24.727188015s) [noise=true , address=145.79.227.11:4731 , piece_id=2BUZ5Q4KA4MG6EBYHXRICRVAE6MBL6QXMJSIZFV66RBCJ7L6MDKQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.203023491 (158h22m24.754493952s) [noise=true , address=45.91.116.14:4190 , piece_id=X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.211675806 (158h22m24.763146267s) [noise=true , address=145.79.227.21:4473 , piece_id=5WUFKUK7GAWMRPMKCGJD3QAXWLE2AH4SP2WN653JYGWZX5OWWGBA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.241230605 (158h22m24.792701066s) [noise=true , address=145.79.227.18:4005 , piece_id=AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.265904560 (158h22m24.817375011s) [noise=true , address=45.91.116.11:4117 , piece_id=SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.267751306 (158h22m24.819221757s) [noise=true , address=145.79.227.27:4691 , piece_id=AN2YAQFX4APSPXZ37L3THXZDLOU3NZJXZX3GFPQ6GOP2XYMYHJMA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.297086917 (158h22m24.848557379s) [noise=true , address=45.91.116.10:4083 , piece_id=A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.396803830 (158h22m24.948274281s) [noise=true , address=77.90.31.12:4592 , piece_id=DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:51.578031970 (158h22m50.129502431s) [noise=true , address=145.79.227.8:4329 , piece_id=C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:51.578058921 (158h22m50.129529382s) [seg_pos={0 0}, upload=139438]: Segment reached optimal threshold of 13 pieces. Nov 15 22:26:51.578335898 (158h22m50.129806369s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Nov 15 22:26:51.578366757 (158h22m50.129837218s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.578860117 (158h22m50.130330588s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection Nov 15 22:26:51.578892328 (158h22m50.130362789s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.579086789 (158h22m50.13055726s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection Nov 15 22:26:51.579099853 (158h22m50.130570314s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.579150771 (158h22m50.130621232s) [seg_pos={0 0}, upload=139438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 22:26:51.579271630 (158h22m50.130742101s) [upload=139438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 22:26:51.579279916 (158h22m50.130750377s) [upload=139438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 22:26:51.887479665 (158h22m50.438950126s) [upload=139438]: Done uploading segments... Nov 15 22:26:51.887500194 (158h22m50.438970655s) [upload=139438]: Upload finished. err: <nil> Duration: 5.949158553s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.926782823s trace: https://jaeger.dev.storj.io/trace/640930c436a3dd40 FAIL (2025-11-15 22:37:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA) context deadline exceeded; piecestore: (Node ID: 12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, Piece ID: AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA) context deadline exceeded; piecestore: (Node ID: 1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, Piece ID: TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA) context deadline exceeded; piecestore: (Node ID: 12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, Piece ID: DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A) context deadline exceeded ======LOGS====== Nov 15 22:26:23.884265358 (158h22m22.43573582s) [upload=139438]: Got next segment. Inline: false Nov 15 22:26:23.884310675 (158h22m22.435781137s) [upload=139438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 22:26:23.884317198 (158h22m22.435787649s) [upload=139438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 22:26:24.145294674 (158h22m22.696765135s) [seg_pos={0 0}, upload=139438]: Begin upload segment... Nov 15 22:26:24.145353216 (158h22m22.696823667s) [seg_pos={0 0}, upload=139438]: Done begin upload segment. Nov 15 22:26:24.145389284 (158h22m22.696859745s) [noise=true , address=145.79.227.28:4727 , piece_id=JTVEZRINCPKC7U532YI62SEXI25AXEE6ZBXEXWGTI4IWVEPTVWSQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.145586640 (158h22m22.697057091s) [noise=true , address=145.79.227.21:4473 , piece_id=5WUFKUK7GAWMRPMKCGJD3QAXWLE2AH4SP2WN653JYGWZX5OWWGBA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146324976 (158h22m22.697795437s) [noise=true , address=145.79.227.11:4731 , piece_id=2BUZ5Q4KA4MG6EBYHXRICRVAE6MBL6QXMJSIZFV66RBCJ7L6MDKQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146351477 (158h22m22.697821938s) [noise=true , address=45.91.116.11:4117 , piece_id=SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146783870 (158h22m22.698254331s) [noise=true , address=145.79.227.18:4005 , piece_id=AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146833304 (158h22m22.698303765s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147216514 (158h22m22.698686975s) [noise=true , address=145.79.227.10:4402 , piece_id=HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147263273 (158h22m22.698733734s) [noise=true , address=45.91.116.14:4190 , piece_id=X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147675318 (158h22m22.699145799s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147754558 (158h22m22.699225019s) [noise=true , address=145.79.227.20:4439 , piece_id=TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148345744 (158h22m22.699816205s) [noise=true , address=45.91.116.10:4083 , piece_id=A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148407592 (158h22m22.699878053s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148798777 (158h22m22.700269228s) [noise=true , address=145.79.227.31:4799 , piece_id=LT3JYVZJ2A3I4VPLCEYONSNIPRBB4E2XSODHWL572IB3NFCM26JA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148858250 (158h22m22.700328711s) [noise=true , address=145.79.227.27:4691 , piece_id=AN2YAQFX4APSPXZ37L3THXZDLOU3NZJXZX3GFPQ6GOP2XYMYHJMA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.149166006 (158h22m22.700636467s) [noise=true , address=145.79.227.8:4329 , piece_id=C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.149373912 (158h22m22.700844373s) [noise=true , address=77.90.31.12:4592 , piece_id=DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:25.877223267 (158h22m24.428693728s) [upload=139438]: Next returned nil segment Nov 15 22:26:25.877245679 (158h22m24.42871614s) [upload=139438]: Waiting for error group managing segments... Nov 15 22:26:26.154633386 (158h22m24.706103847s) [noise=true , address=145.79.227.31:4799 , piece_id=LT3JYVZJ2A3I4VPLCEYONSNIPRBB4E2XSODHWL572IB3NFCM26JA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.172371583 (158h22m24.723842044s) [noise=true , address=145.79.227.28:4727 , piece_id=JTVEZRINCPKC7U532YI62SEXI25AXEE6ZBXEXWGTI4IWVEPTVWSQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.173052429 (158h22m24.72452289s) [noise=true , address=145.79.227.20:4439 , piece_id=TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.174665170 (158h22m24.726135621s) [noise=true , address=145.79.227.10:4402 , piece_id=HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.175717564 (158h22m24.727188015s) [noise=true , address=145.79.227.11:4731 , piece_id=2BUZ5Q4KA4MG6EBYHXRICRVAE6MBL6QXMJSIZFV66RBCJ7L6MDKQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.203023491 (158h22m24.754493952s) [noise=true , address=45.91.116.14:4190 , piece_id=X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.211675806 (158h22m24.763146267s) [noise=true , address=145.79.227.21:4473 , piece_id=5WUFKUK7GAWMRPMKCGJD3QAXWLE2AH4SP2WN653JYGWZX5OWWGBA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.241230605 (158h22m24.792701066s) [noise=true , address=145.79.227.18:4005 , piece_id=AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.265904560 (158h22m24.817375011s) [noise=true , address=45.91.116.11:4117 , piece_id=SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.267751306 (158h22m24.819221757s) [noise=true , address=145.79.227.27:4691 , piece_id=AN2YAQFX4APSPXZ37L3THXZDLOU3NZJXZX3GFPQ6GOP2XYMYHJMA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.297086917 (158h22m24.848557379s) [noise=true , address=45.91.116.10:4083 , piece_id=A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.396803830 (158h22m24.948274281s) [noise=true , address=77.90.31.12:4592 , piece_id=DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:51.578031970 (158h22m50.129502431s) [noise=true , address=145.79.227.8:4329 , piece_id=C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:51.578058921 (158h22m50.129529382s) [seg_pos={0 0}, upload=139438]: Segment reached optimal threshold of 13 pieces. Nov 15 22:26:51.578335898 (158h22m50.129806369s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Nov 15 22:26:51.578366757 (158h22m50.129837218s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.578860117 (158h22m50.130330588s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection Nov 15 22:26:51.578892328 (158h22m50.130362789s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.579086789 (158h22m50.13055726s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection Nov 15 22:26:51.579099853 (158h22m50.130570314s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.579150771 (158h22m50.130621232s) [seg_pos={0 0}, upload=139438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 22:26:51.579271630 (158h22m50.130742101s) [upload=139438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 22:26:51.579279916 (158h22m50.130750377s) [upload=139438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 22:26:51.887479665 (158h22m50.438950126s) [upload=139438]: Done uploading segments... Nov 15 22:26:51.887500194 (158h22m50.438970655s) [upload=139438]: Upload finished. err: <nil> Duration: 5.9268129s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.112862315s trace: https://jaeger.dev.storj.io/trace/640930c436a3dd40 FAIL (2025-11-15 22:37:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA) context deadline exceeded; piecestore: (Node ID: 12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, Piece ID: AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA) context deadline exceeded; piecestore: (Node ID: 1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, Piece ID: TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA) context deadline exceeded; piecestore: (Node ID: 12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, Piece ID: DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A) context deadline exceeded ======LOGS====== Nov 15 22:26:23.884265358 (158h22m22.43573582s) [upload=139438]: Got next segment. Inline: false Nov 15 22:26:23.884310675 (158h22m22.435781137s) [upload=139438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 22:26:23.884317198 (158h22m22.435787649s) [upload=139438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 22:26:24.145294674 (158h22m22.696765135s) [seg_pos={0 0}, upload=139438]: Begin upload segment... Nov 15 22:26:24.145353216 (158h22m22.696823667s) [seg_pos={0 0}, upload=139438]: Done begin upload segment. Nov 15 22:26:24.145389284 (158h22m22.696859745s) [noise=true , address=145.79.227.28:4727 , piece_id=JTVEZRINCPKC7U532YI62SEXI25AXEE6ZBXEXWGTI4IWVEPTVWSQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.145586640 (158h22m22.697057091s) [noise=true , address=145.79.227.21:4473 , piece_id=5WUFKUK7GAWMRPMKCGJD3QAXWLE2AH4SP2WN653JYGWZX5OWWGBA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146324976 (158h22m22.697795437s) [noise=true , address=145.79.227.11:4731 , piece_id=2BUZ5Q4KA4MG6EBYHXRICRVAE6MBL6QXMJSIZFV66RBCJ7L6MDKQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146351477 (158h22m22.697821938s) [noise=true , address=45.91.116.11:4117 , piece_id=SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146783870 (158h22m22.698254331s) [noise=true , address=145.79.227.18:4005 , piece_id=AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146833304 (158h22m22.698303765s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147216514 (158h22m22.698686975s) [noise=true , address=145.79.227.10:4402 , piece_id=HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147263273 (158h22m22.698733734s) [noise=true , address=45.91.116.14:4190 , piece_id=X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147675318 (158h22m22.699145799s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147754558 (158h22m22.699225019s) [noise=true , address=145.79.227.20:4439 , piece_id=TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148345744 (158h22m22.699816205s) [noise=true , address=45.91.116.10:4083 , piece_id=A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148407592 (158h22m22.699878053s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148798777 (158h22m22.700269228s) [noise=true , address=145.79.227.31:4799 , piece_id=LT3JYVZJ2A3I4VPLCEYONSNIPRBB4E2XSODHWL572IB3NFCM26JA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148858250 (158h22m22.700328711s) [noise=true , address=145.79.227.27:4691 , piece_id=AN2YAQFX4APSPXZ37L3THXZDLOU3NZJXZX3GFPQ6GOP2XYMYHJMA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.149166006 (158h22m22.700636467s) [noise=true , address=145.79.227.8:4329 , piece_id=C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.149373912 (158h22m22.700844373s) [noise=true , address=77.90.31.12:4592 , piece_id=DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:25.877223267 (158h22m24.428693728s) [upload=139438]: Next returned nil segment Nov 15 22:26:25.877245679 (158h22m24.42871614s) [upload=139438]: Waiting for error group managing segments... Nov 15 22:26:26.154633386 (158h22m24.706103847s) [noise=true , address=145.79.227.31:4799 , piece_id=LT3JYVZJ2A3I4VPLCEYONSNIPRBB4E2XSODHWL572IB3NFCM26JA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.172371583 (158h22m24.723842044s) [noise=true , address=145.79.227.28:4727 , piece_id=JTVEZRINCPKC7U532YI62SEXI25AXEE6ZBXEXWGTI4IWVEPTVWSQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.173052429 (158h22m24.72452289s) [noise=true , address=145.79.227.20:4439 , piece_id=TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.174665170 (158h22m24.726135621s) [noise=true , address=145.79.227.10:4402 , piece_id=HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.175717564 (158h22m24.727188015s) [noise=true , address=145.79.227.11:4731 , piece_id=2BUZ5Q4KA4MG6EBYHXRICRVAE6MBL6QXMJSIZFV66RBCJ7L6MDKQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.203023491 (158h22m24.754493952s) [noise=true , address=45.91.116.14:4190 , piece_id=X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.211675806 (158h22m24.763146267s) [noise=true , address=145.79.227.21:4473 , piece_id=5WUFKUK7GAWMRPMKCGJD3QAXWLE2AH4SP2WN653JYGWZX5OWWGBA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.241230605 (158h22m24.792701066s) [noise=true , address=145.79.227.18:4005 , piece_id=AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.265904560 (158h22m24.817375011s) [noise=true , address=45.91.116.11:4117 , piece_id=SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.267751306 (158h22m24.819221757s) [noise=true , address=145.79.227.27:4691 , piece_id=AN2YAQFX4APSPXZ37L3THXZDLOU3NZJXZX3GFPQ6GOP2XYMYHJMA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.297086917 (158h22m24.848557379s) [noise=true , address=45.91.116.10:4083 , piece_id=A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.396803830 (158h22m24.948274281s) [noise=true , address=77.90.31.12:4592 , piece_id=DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:51.578031970 (158h22m50.129502431s) [noise=true , address=145.79.227.8:4329 , piece_id=C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:51.578058921 (158h22m50.129529382s) [seg_pos={0 0}, upload=139438]: Segment reached optimal threshold of 13 pieces. Nov 15 22:26:51.578335898 (158h22m50.129806369s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Nov 15 22:26:51.578366757 (158h22m50.129837218s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.578860117 (158h22m50.130330588s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection Nov 15 22:26:51.578892328 (158h22m50.130362789s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.579086789 (158h22m50.13055726s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection Nov 15 22:26:51.579099853 (158h22m50.130570314s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.579150771 (158h22m50.130621232s) [seg_pos={0 0}, upload=139438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 22:26:51.579271630 (158h22m50.130742101s) [upload=139438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 22:26:51.579279916 (158h22m50.130750377s) [upload=139438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 22:26:51.887479665 (158h22m50.438950126s) [upload=139438]: Done uploading segments... Nov 15 22:26:51.887500194 (158h22m50.438970655s) [upload=139438]: Upload finished. err: <nil> Duration: 6.112897202s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.238517522s trace: https://jaeger.dev.storj.io/trace/640930c436a3dd40 FAIL (2025-11-15 22:36:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA) context deadline exceeded; piecestore: (Node ID: 12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, Piece ID: AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA) context deadline exceeded; piecestore: (Node ID: 1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, Piece ID: TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA) context deadline exceeded; piecestore: (Node ID: 12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, Piece ID: DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A) context deadline exceeded ======LOGS====== Nov 15 22:26:23.884265358 (158h22m22.43573582s) [upload=139438]: Got next segment. Inline: false Nov 15 22:26:23.884310675 (158h22m22.435781137s) [upload=139438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 22:26:23.884317198 (158h22m22.435787649s) [upload=139438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 22:26:24.145294674 (158h22m22.696765135s) [seg_pos={0 0}, upload=139438]: Begin upload segment... Nov 15 22:26:24.145353216 (158h22m22.696823667s) [seg_pos={0 0}, upload=139438]: Done begin upload segment. Nov 15 22:26:24.145389284 (158h22m22.696859745s) [noise=true , address=145.79.227.28:4727 , piece_id=JTVEZRINCPKC7U532YI62SEXI25AXEE6ZBXEXWGTI4IWVEPTVWSQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.145586640 (158h22m22.697057091s) [noise=true , address=145.79.227.21:4473 , piece_id=5WUFKUK7GAWMRPMKCGJD3QAXWLE2AH4SP2WN653JYGWZX5OWWGBA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146324976 (158h22m22.697795437s) [noise=true , address=145.79.227.11:4731 , piece_id=2BUZ5Q4KA4MG6EBYHXRICRVAE6MBL6QXMJSIZFV66RBCJ7L6MDKQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146351477 (158h22m22.697821938s) [noise=true , address=45.91.116.11:4117 , piece_id=SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146783870 (158h22m22.698254331s) [noise=true , address=145.79.227.18:4005 , piece_id=AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146833304 (158h22m22.698303765s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147216514 (158h22m22.698686975s) [noise=true , address=145.79.227.10:4402 , piece_id=HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147263273 (158h22m22.698733734s) [noise=true , address=45.91.116.14:4190 , piece_id=X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147675318 (158h22m22.699145799s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147754558 (158h22m22.699225019s) [noise=true , address=145.79.227.20:4439 , piece_id=TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148345744 (158h22m22.699816205s) [noise=true , address=45.91.116.10:4083 , piece_id=A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148407592 (158h22m22.699878053s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148798777 (158h22m22.700269228s) [noise=true , address=145.79.227.31:4799 , piece_id=LT3JYVZJ2A3I4VPLCEYONSNIPRBB4E2XSODHWL572IB3NFCM26JA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148858250 (158h22m22.700328711s) [noise=true , address=145.79.227.27:4691 , piece_id=AN2YAQFX4APSPXZ37L3THXZDLOU3NZJXZX3GFPQ6GOP2XYMYHJMA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.149166006 (158h22m22.700636467s) [noise=true , address=145.79.227.8:4329 , piece_id=C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.149373912 (158h22m22.700844373s) [noise=true , address=77.90.31.12:4592 , piece_id=DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:25.877223267 (158h22m24.428693728s) [upload=139438]: Next returned nil segment Nov 15 22:26:25.877245679 (158h22m24.42871614s) [upload=139438]: Waiting for error group managing segments... Nov 15 22:26:26.154633386 (158h22m24.706103847s) [noise=true , address=145.79.227.31:4799 , piece_id=LT3JYVZJ2A3I4VPLCEYONSNIPRBB4E2XSODHWL572IB3NFCM26JA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.172371583 (158h22m24.723842044s) [noise=true , address=145.79.227.28:4727 , piece_id=JTVEZRINCPKC7U532YI62SEXI25AXEE6ZBXEXWGTI4IWVEPTVWSQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.173052429 (158h22m24.72452289s) [noise=true , address=145.79.227.20:4439 , piece_id=TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.174665170 (158h22m24.726135621s) [noise=true , address=145.79.227.10:4402 , piece_id=HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.175717564 (158h22m24.727188015s) [noise=true , address=145.79.227.11:4731 , piece_id=2BUZ5Q4KA4MG6EBYHXRICRVAE6MBL6QXMJSIZFV66RBCJ7L6MDKQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.203023491 (158h22m24.754493952s) [noise=true , address=45.91.116.14:4190 , piece_id=X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.211675806 (158h22m24.763146267s) [noise=true , address=145.79.227.21:4473 , piece_id=5WUFKUK7GAWMRPMKCGJD3QAXWLE2AH4SP2WN653JYGWZX5OWWGBA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.241230605 (158h22m24.792701066s) [noise=true , address=145.79.227.18:4005 , piece_id=AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.265904560 (158h22m24.817375011s) [noise=true , address=45.91.116.11:4117 , piece_id=SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.267751306 (158h22m24.819221757s) [noise=true , address=145.79.227.27:4691 , piece_id=AN2YAQFX4APSPXZ37L3THXZDLOU3NZJXZX3GFPQ6GOP2XYMYHJMA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.297086917 (158h22m24.848557379s) [noise=true , address=45.91.116.10:4083 , piece_id=A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.396803830 (158h22m24.948274281s) [noise=true , address=77.90.31.12:4592 , piece_id=DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:51.578031970 (158h22m50.129502431s) [noise=true , address=145.79.227.8:4329 , piece_id=C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:51.578058921 (158h22m50.129529382s) [seg_pos={0 0}, upload=139438]: Segment reached optimal threshold of 13 pieces. Nov 15 22:26:51.578335898 (158h22m50.129806369s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Nov 15 22:26:51.578366757 (158h22m50.129837218s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.578860117 (158h22m50.130330588s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection Nov 15 22:26:51.578892328 (158h22m50.130362789s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.579086789 (158h22m50.13055726s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection Nov 15 22:26:51.579099853 (158h22m50.130570314s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.579150771 (158h22m50.130621232s) [seg_pos={0 0}, upload=139438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 22:26:51.579271630 (158h22m50.130742101s) [upload=139438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 22:26:51.579279916 (158h22m50.130750377s) [upload=139438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 22:26:51.887479665 (158h22m50.438950126s) [upload=139438]: Done uploading segments... Nov 15 22:26:51.887500194 (158h22m50.438970655s) [upload=139438]: Upload finished. err: <nil> Duration: 6.238546817s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.844955672s trace: https://jaeger.dev.storj.io/trace/640930c436a3dd40 FAIL (2025-11-15 22:35:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA) context deadline exceeded; piecestore: (Node ID: 12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, Piece ID: AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA) context deadline exceeded; piecestore: (Node ID: 1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, Piece ID: TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA) context deadline exceeded; piecestore: (Node ID: 12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, Piece ID: DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A) context deadline exceeded ======LOGS====== Nov 15 22:26:23.884265358 (158h22m22.43573582s) [upload=139438]: Got next segment. Inline: false Nov 15 22:26:23.884310675 (158h22m22.435781137s) [upload=139438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 22:26:23.884317198 (158h22m22.435787649s) [upload=139438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 22:26:24.145294674 (158h22m22.696765135s) [seg_pos={0 0}, upload=139438]: Begin upload segment... Nov 15 22:26:24.145353216 (158h22m22.696823667s) [seg_pos={0 0}, upload=139438]: Done begin upload segment. Nov 15 22:26:24.145389284 (158h22m22.696859745s) [noise=true , address=145.79.227.28:4727 , piece_id=JTVEZRINCPKC7U532YI62SEXI25AXEE6ZBXEXWGTI4IWVEPTVWSQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.145586640 (158h22m22.697057091s) [noise=true , address=145.79.227.21:4473 , piece_id=5WUFKUK7GAWMRPMKCGJD3QAXWLE2AH4SP2WN653JYGWZX5OWWGBA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146324976 (158h22m22.697795437s) [noise=true , address=145.79.227.11:4731 , piece_id=2BUZ5Q4KA4MG6EBYHXRICRVAE6MBL6QXMJSIZFV66RBCJ7L6MDKQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146351477 (158h22m22.697821938s) [noise=true , address=45.91.116.11:4117 , piece_id=SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146783870 (158h22m22.698254331s) [noise=true , address=145.79.227.18:4005 , piece_id=AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146833304 (158h22m22.698303765s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147216514 (158h22m22.698686975s) [noise=true , address=145.79.227.10:4402 , piece_id=HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147263273 (158h22m22.698733734s) [noise=true , address=45.91.116.14:4190 , piece_id=X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147675318 (158h22m22.699145799s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147754558 (158h22m22.699225019s) [noise=true , address=145.79.227.20:4439 , piece_id=TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148345744 (158h22m22.699816205s) [noise=true , address=45.91.116.10:4083 , piece_id=A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148407592 (158h22m22.699878053s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148798777 (158h22m22.700269228s) [noise=true , address=145.79.227.31:4799 , piece_id=LT3JYVZJ2A3I4VPLCEYONSNIPRBB4E2XSODHWL572IB3NFCM26JA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148858250 (158h22m22.700328711s) [noise=true , address=145.79.227.27:4691 , piece_id=AN2YAQFX4APSPXZ37L3THXZDLOU3NZJXZX3GFPQ6GOP2XYMYHJMA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.149166006 (158h22m22.700636467s) [noise=true , address=145.79.227.8:4329 , piece_id=C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.149373912 (158h22m22.700844373s) [noise=true , address=77.90.31.12:4592 , piece_id=DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:25.877223267 (158h22m24.428693728s) [upload=139438]: Next returned nil segment Nov 15 22:26:25.877245679 (158h22m24.42871614s) [upload=139438]: Waiting for error group managing segments... Nov 15 22:26:26.154633386 (158h22m24.706103847s) [noise=true , address=145.79.227.31:4799 , piece_id=LT3JYVZJ2A3I4VPLCEYONSNIPRBB4E2XSODHWL572IB3NFCM26JA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.172371583 (158h22m24.723842044s) [noise=true , address=145.79.227.28:4727 , piece_id=JTVEZRINCPKC7U532YI62SEXI25AXEE6ZBXEXWGTI4IWVEPTVWSQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.173052429 (158h22m24.72452289s) [noise=true , address=145.79.227.20:4439 , piece_id=TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.174665170 (158h22m24.726135621s) [noise=true , address=145.79.227.10:4402 , piece_id=HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.175717564 (158h22m24.727188015s) [noise=true , address=145.79.227.11:4731 , piece_id=2BUZ5Q4KA4MG6EBYHXRICRVAE6MBL6QXMJSIZFV66RBCJ7L6MDKQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.203023491 (158h22m24.754493952s) [noise=true , address=45.91.116.14:4190 , piece_id=X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.211675806 (158h22m24.763146267s) [noise=true , address=145.79.227.21:4473 , piece_id=5WUFKUK7GAWMRPMKCGJD3QAXWLE2AH4SP2WN653JYGWZX5OWWGBA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.241230605 (158h22m24.792701066s) [noise=true , address=145.79.227.18:4005 , piece_id=AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.265904560 (158h22m24.817375011s) [noise=true , address=45.91.116.11:4117 , piece_id=SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.267751306 (158h22m24.819221757s) [noise=true , address=145.79.227.27:4691 , piece_id=AN2YAQFX4APSPXZ37L3THXZDLOU3NZJXZX3GFPQ6GOP2XYMYHJMA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.297086917 (158h22m24.848557379s) [noise=true , address=45.91.116.10:4083 , piece_id=A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.396803830 (158h22m24.948274281s) [noise=true , address=77.90.31.12:4592 , piece_id=DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:51.578031970 (158h22m50.129502431s) [noise=true , address=145.79.227.8:4329 , piece_id=C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:51.578058921 (158h22m50.129529382s) [seg_pos={0 0}, upload=139438]: Segment reached optimal threshold of 13 pieces. Nov 15 22:26:51.578335898 (158h22m50.129806369s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Nov 15 22:26:51.578366757 (158h22m50.129837218s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.578860117 (158h22m50.130330588s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection Nov 15 22:26:51.578892328 (158h22m50.130362789s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.579086789 (158h22m50.13055726s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection Nov 15 22:26:51.579099853 (158h22m50.130570314s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.579150771 (158h22m50.130621232s) [seg_pos={0 0}, upload=139438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 22:26:51.579271630 (158h22m50.130742101s) [upload=139438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 22:26:51.579279916 (158h22m50.130750377s) [upload=139438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 22:26:51.887479665 (158h22m50.438950126s) [upload=139438]: Done uploading segments... Nov 15 22:26:51.887500194 (158h22m50.438970655s) [upload=139438]: Upload finished. err: <nil> Duration: 5.84498623s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.919933099s trace: https://jaeger.dev.storj.io/trace/640930c436a3dd40 FAIL (2025-11-15 22:34:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA) context deadline exceeded; piecestore: (Node ID: 12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, Piece ID: AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA) context deadline exceeded; piecestore: (Node ID: 1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, Piece ID: TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA) context deadline exceeded; piecestore: (Node ID: 12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, Piece ID: DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A) context deadline exceeded ======LOGS====== Nov 15 22:26:23.884265358 (158h22m22.43573582s) [upload=139438]: Got next segment. Inline: false Nov 15 22:26:23.884310675 (158h22m22.435781137s) [upload=139438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 22:26:23.884317198 (158h22m22.435787649s) [upload=139438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 22:26:24.145294674 (158h22m22.696765135s) [seg_pos={0 0}, upload=139438]: Begin upload segment... Nov 15 22:26:24.145353216 (158h22m22.696823667s) [seg_pos={0 0}, upload=139438]: Done begin upload segment. Nov 15 22:26:24.145389284 (158h22m22.696859745s) [noise=true , address=145.79.227.28:4727 , piece_id=JTVEZRINCPKC7U532YI62SEXI25AXEE6ZBXEXWGTI4IWVEPTVWSQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.145586640 (158h22m22.697057091s) [noise=true , address=145.79.227.21:4473 , piece_id=5WUFKUK7GAWMRPMKCGJD3QAXWLE2AH4SP2WN653JYGWZX5OWWGBA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146324976 (158h22m22.697795437s) [noise=true , address=145.79.227.11:4731 , piece_id=2BUZ5Q4KA4MG6EBYHXRICRVAE6MBL6QXMJSIZFV66RBCJ7L6MDKQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146351477 (158h22m22.697821938s) [noise=true , address=45.91.116.11:4117 , piece_id=SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146783870 (158h22m22.698254331s) [noise=true , address=145.79.227.18:4005 , piece_id=AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146833304 (158h22m22.698303765s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147216514 (158h22m22.698686975s) [noise=true , address=145.79.227.10:4402 , piece_id=HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147263273 (158h22m22.698733734s) [noise=true , address=45.91.116.14:4190 , piece_id=X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147675318 (158h22m22.699145799s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147754558 (158h22m22.699225019s) [noise=true , address=145.79.227.20:4439 , piece_id=TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148345744 (158h22m22.699816205s) [noise=true , address=45.91.116.10:4083 , piece_id=A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148407592 (158h22m22.699878053s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148798777 (158h22m22.700269228s) [noise=true , address=145.79.227.31:4799 , piece_id=LT3JYVZJ2A3I4VPLCEYONSNIPRBB4E2XSODHWL572IB3NFCM26JA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148858250 (158h22m22.700328711s) [noise=true , address=145.79.227.27:4691 , piece_id=AN2YAQFX4APSPXZ37L3THXZDLOU3NZJXZX3GFPQ6GOP2XYMYHJMA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.149166006 (158h22m22.700636467s) [noise=true , address=145.79.227.8:4329 , piece_id=C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.149373912 (158h22m22.700844373s) [noise=true , address=77.90.31.12:4592 , piece_id=DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:25.877223267 (158h22m24.428693728s) [upload=139438]: Next returned nil segment Nov 15 22:26:25.877245679 (158h22m24.42871614s) [upload=139438]: Waiting for error group managing segments... Nov 15 22:26:26.154633386 (158h22m24.706103847s) [noise=true , address=145.79.227.31:4799 , piece_id=LT3JYVZJ2A3I4VPLCEYONSNIPRBB4E2XSODHWL572IB3NFCM26JA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.172371583 (158h22m24.723842044s) [noise=true , address=145.79.227.28:4727 , piece_id=JTVEZRINCPKC7U532YI62SEXI25AXEE6ZBXEXWGTI4IWVEPTVWSQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.173052429 (158h22m24.72452289s) [noise=true , address=145.79.227.20:4439 , piece_id=TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.174665170 (158h22m24.726135621s) [noise=true , address=145.79.227.10:4402 , piece_id=HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.175717564 (158h22m24.727188015s) [noise=true , address=145.79.227.11:4731 , piece_id=2BUZ5Q4KA4MG6EBYHXRICRVAE6MBL6QXMJSIZFV66RBCJ7L6MDKQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.203023491 (158h22m24.754493952s) [noise=true , address=45.91.116.14:4190 , piece_id=X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.211675806 (158h22m24.763146267s) [noise=true , address=145.79.227.21:4473 , piece_id=5WUFKUK7GAWMRPMKCGJD3QAXWLE2AH4SP2WN653JYGWZX5OWWGBA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.241230605 (158h22m24.792701066s) [noise=true , address=145.79.227.18:4005 , piece_id=AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.265904560 (158h22m24.817375011s) [noise=true , address=45.91.116.11:4117 , piece_id=SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.267751306 (158h22m24.819221757s) [noise=true , address=145.79.227.27:4691 , piece_id=AN2YAQFX4APSPXZ37L3THXZDLOU3NZJXZX3GFPQ6GOP2XYMYHJMA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.297086917 (158h22m24.848557379s) [noise=true , address=45.91.116.10:4083 , piece_id=A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.396803830 (158h22m24.948274281s) [noise=true , address=77.90.31.12:4592 , piece_id=DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:51.578031970 (158h22m50.129502431s) [noise=true , address=145.79.227.8:4329 , piece_id=C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:51.578058921 (158h22m50.129529382s) [seg_pos={0 0}, upload=139438]: Segment reached optimal threshold of 13 pieces. Nov 15 22:26:51.578335898 (158h22m50.129806369s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Nov 15 22:26:51.578366757 (158h22m50.129837218s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.578860117 (158h22m50.130330588s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection Nov 15 22:26:51.578892328 (158h22m50.130362789s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.579086789 (158h22m50.13055726s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection Nov 15 22:26:51.579099853 (158h22m50.130570314s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.579150771 (158h22m50.130621232s) [seg_pos={0 0}, upload=139438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 22:26:51.579271630 (158h22m50.130742101s) [upload=139438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 22:26:51.579279916 (158h22m50.130750377s) [upload=139438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 22:26:51.887479665 (158h22m50.438950126s) [upload=139438]: Done uploading segments... Nov 15 22:26:51.887500194 (158h22m50.438970655s) [upload=139438]: Upload finished. err: <nil> Duration: 5.919963115s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.02673583s trace: https://jaeger.dev.storj.io/trace/640930c436a3dd40 FAIL (2025-11-15 22:33:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA) context deadline exceeded; piecestore: (Node ID: 12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, Piece ID: AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA) context deadline exceeded; piecestore: (Node ID: 1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, Piece ID: TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA) context deadline exceeded; piecestore: (Node ID: 12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, Piece ID: DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A) context deadline exceeded ======LOGS====== Nov 15 22:26:23.884265358 (158h22m22.43573582s) [upload=139438]: Got next segment. Inline: false Nov 15 22:26:23.884310675 (158h22m22.435781137s) [upload=139438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 22:26:23.884317198 (158h22m22.435787649s) [upload=139438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 22:26:24.145294674 (158h22m22.696765135s) [seg_pos={0 0}, upload=139438]: Begin upload segment... Nov 15 22:26:24.145353216 (158h22m22.696823667s) [seg_pos={0 0}, upload=139438]: Done begin upload segment. Nov 15 22:26:24.145389284 (158h22m22.696859745s) [noise=true , address=145.79.227.28:4727 , piece_id=JTVEZRINCPKC7U532YI62SEXI25AXEE6ZBXEXWGTI4IWVEPTVWSQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.145586640 (158h22m22.697057091s) [noise=true , address=145.79.227.21:4473 , piece_id=5WUFKUK7GAWMRPMKCGJD3QAXWLE2AH4SP2WN653JYGWZX5OWWGBA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146324976 (158h22m22.697795437s) [noise=true , address=145.79.227.11:4731 , piece_id=2BUZ5Q4KA4MG6EBYHXRICRVAE6MBL6QXMJSIZFV66RBCJ7L6MDKQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146351477 (158h22m22.697821938s) [noise=true , address=45.91.116.11:4117 , piece_id=SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146783870 (158h22m22.698254331s) [noise=true , address=145.79.227.18:4005 , piece_id=AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146833304 (158h22m22.698303765s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147216514 (158h22m22.698686975s) [noise=true , address=145.79.227.10:4402 , piece_id=HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147263273 (158h22m22.698733734s) [noise=true , address=45.91.116.14:4190 , piece_id=X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147675318 (158h22m22.699145799s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147754558 (158h22m22.699225019s) [noise=true , address=145.79.227.20:4439 , piece_id=TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148345744 (158h22m22.699816205s) [noise=true , address=45.91.116.10:4083 , piece_id=A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148407592 (158h22m22.699878053s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148798777 (158h22m22.700269228s) [noise=true , address=145.79.227.31:4799 , piece_id=LT3JYVZJ2A3I4VPLCEYONSNIPRBB4E2XSODHWL572IB3NFCM26JA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148858250 (158h22m22.700328711s) [noise=true , address=145.79.227.27:4691 , piece_id=AN2YAQFX4APSPXZ37L3THXZDLOU3NZJXZX3GFPQ6GOP2XYMYHJMA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.149166006 (158h22m22.700636467s) [noise=true , address=145.79.227.8:4329 , piece_id=C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.149373912 (158h22m22.700844373s) [noise=true , address=77.90.31.12:4592 , piece_id=DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:25.877223267 (158h22m24.428693728s) [upload=139438]: Next returned nil segment Nov 15 22:26:25.877245679 (158h22m24.42871614s) [upload=139438]: Waiting for error group managing segments... Nov 15 22:26:26.154633386 (158h22m24.706103847s) [noise=true , address=145.79.227.31:4799 , piece_id=LT3JYVZJ2A3I4VPLCEYONSNIPRBB4E2XSODHWL572IB3NFCM26JA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.172371583 (158h22m24.723842044s) [noise=true , address=145.79.227.28:4727 , piece_id=JTVEZRINCPKC7U532YI62SEXI25AXEE6ZBXEXWGTI4IWVEPTVWSQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.173052429 (158h22m24.72452289s) [noise=true , address=145.79.227.20:4439 , piece_id=TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.174665170 (158h22m24.726135621s) [noise=true , address=145.79.227.10:4402 , piece_id=HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.175717564 (158h22m24.727188015s) [noise=true , address=145.79.227.11:4731 , piece_id=2BUZ5Q4KA4MG6EBYHXRICRVAE6MBL6QXMJSIZFV66RBCJ7L6MDKQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.203023491 (158h22m24.754493952s) [noise=true , address=45.91.116.14:4190 , piece_id=X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.211675806 (158h22m24.763146267s) [noise=true , address=145.79.227.21:4473 , piece_id=5WUFKUK7GAWMRPMKCGJD3QAXWLE2AH4SP2WN653JYGWZX5OWWGBA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.241230605 (158h22m24.792701066s) [noise=true , address=145.79.227.18:4005 , piece_id=AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.265904560 (158h22m24.817375011s) [noise=true , address=45.91.116.11:4117 , piece_id=SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.267751306 (158h22m24.819221757s) [noise=true , address=145.79.227.27:4691 , piece_id=AN2YAQFX4APSPXZ37L3THXZDLOU3NZJXZX3GFPQ6GOP2XYMYHJMA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.297086917 (158h22m24.848557379s) [noise=true , address=45.91.116.10:4083 , piece_id=A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.396803830 (158h22m24.948274281s) [noise=true , address=77.90.31.12:4592 , piece_id=DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:51.578031970 (158h22m50.129502431s) [noise=true , address=145.79.227.8:4329 , piece_id=C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:51.578058921 (158h22m50.129529382s) [seg_pos={0 0}, upload=139438]: Segment reached optimal threshold of 13 pieces. Nov 15 22:26:51.578335898 (158h22m50.129806369s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Nov 15 22:26:51.578366757 (158h22m50.129837218s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.578860117 (158h22m50.130330588s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection Nov 15 22:26:51.578892328 (158h22m50.130362789s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.579086789 (158h22m50.13055726s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection Nov 15 22:26:51.579099853 (158h22m50.130570314s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.579150771 (158h22m50.130621232s) [seg_pos={0 0}, upload=139438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 22:26:51.579271630 (158h22m50.130742101s) [upload=139438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 22:26:51.579279916 (158h22m50.130750377s) [upload=139438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 22:26:51.887479665 (158h22m50.438950126s) [upload=139438]: Done uploading segments... Nov 15 22:26:51.887500194 (158h22m50.438970655s) [upload=139438]: Upload finished. err: <nil> Duration: 6.026770335s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.997536027s trace: https://jaeger.dev.storj.io/trace/640930c436a3dd40 FAIL (2025-11-15 22:32:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA) context deadline exceeded; piecestore: (Node ID: 12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, Piece ID: AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA) context deadline exceeded; piecestore: (Node ID: 1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, Piece ID: TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA) context deadline exceeded; piecestore: (Node ID: 12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, Piece ID: DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A) context deadline exceeded ======LOGS====== Nov 15 22:26:23.884265358 (158h22m22.43573582s) [upload=139438]: Got next segment. Inline: false Nov 15 22:26:23.884310675 (158h22m22.435781137s) [upload=139438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 22:26:23.884317198 (158h22m22.435787649s) [upload=139438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 22:26:24.145294674 (158h22m22.696765135s) [seg_pos={0 0}, upload=139438]: Begin upload segment... Nov 15 22:26:24.145353216 (158h22m22.696823667s) [seg_pos={0 0}, upload=139438]: Done begin upload segment. Nov 15 22:26:24.145389284 (158h22m22.696859745s) [noise=true , address=145.79.227.28:4727 , piece_id=JTVEZRINCPKC7U532YI62SEXI25AXEE6ZBXEXWGTI4IWVEPTVWSQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.145586640 (158h22m22.697057091s) [noise=true , address=145.79.227.21:4473 , piece_id=5WUFKUK7GAWMRPMKCGJD3QAXWLE2AH4SP2WN653JYGWZX5OWWGBA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146324976 (158h22m22.697795437s) [noise=true , address=145.79.227.11:4731 , piece_id=2BUZ5Q4KA4MG6EBYHXRICRVAE6MBL6QXMJSIZFV66RBCJ7L6MDKQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146351477 (158h22m22.697821938s) [noise=true , address=45.91.116.11:4117 , piece_id=SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146783870 (158h22m22.698254331s) [noise=true , address=145.79.227.18:4005 , piece_id=AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146833304 (158h22m22.698303765s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147216514 (158h22m22.698686975s) [noise=true , address=145.79.227.10:4402 , piece_id=HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147263273 (158h22m22.698733734s) [noise=true , address=45.91.116.14:4190 , piece_id=X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147675318 (158h22m22.699145799s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147754558 (158h22m22.699225019s) [noise=true , address=145.79.227.20:4439 , piece_id=TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148345744 (158h22m22.699816205s) [noise=true , address=45.91.116.10:4083 , piece_id=A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148407592 (158h22m22.699878053s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148798777 (158h22m22.700269228s) [noise=true , address=145.79.227.31:4799 , piece_id=LT3JYVZJ2A3I4VPLCEYONSNIPRBB4E2XSODHWL572IB3NFCM26JA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148858250 (158h22m22.700328711s) [noise=true , address=145.79.227.27:4691 , piece_id=AN2YAQFX4APSPXZ37L3THXZDLOU3NZJXZX3GFPQ6GOP2XYMYHJMA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.149166006 (158h22m22.700636467s) [noise=true , address=145.79.227.8:4329 , piece_id=C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.149373912 (158h22m22.700844373s) [noise=true , address=77.90.31.12:4592 , piece_id=DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:25.877223267 (158h22m24.428693728s) [upload=139438]: Next returned nil segment Nov 15 22:26:25.877245679 (158h22m24.42871614s) [upload=139438]: Waiting for error group managing segments... Nov 15 22:26:26.154633386 (158h22m24.706103847s) [noise=true , address=145.79.227.31:4799 , piece_id=LT3JYVZJ2A3I4VPLCEYONSNIPRBB4E2XSODHWL572IB3NFCM26JA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.172371583 (158h22m24.723842044s) [noise=true , address=145.79.227.28:4727 , piece_id=JTVEZRINCPKC7U532YI62SEXI25AXEE6ZBXEXWGTI4IWVEPTVWSQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.173052429 (158h22m24.72452289s) [noise=true , address=145.79.227.20:4439 , piece_id=TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.174665170 (158h22m24.726135621s) [noise=true , address=145.79.227.10:4402 , piece_id=HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.175717564 (158h22m24.727188015s) [noise=true , address=145.79.227.11:4731 , piece_id=2BUZ5Q4KA4MG6EBYHXRICRVAE6MBL6QXMJSIZFV66RBCJ7L6MDKQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.203023491 (158h22m24.754493952s) [noise=true , address=45.91.116.14:4190 , piece_id=X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.211675806 (158h22m24.763146267s) [noise=true , address=145.79.227.21:4473 , piece_id=5WUFKUK7GAWMRPMKCGJD3QAXWLE2AH4SP2WN653JYGWZX5OWWGBA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.241230605 (158h22m24.792701066s) [noise=true , address=145.79.227.18:4005 , piece_id=AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.265904560 (158h22m24.817375011s) [noise=true , address=45.91.116.11:4117 , piece_id=SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.267751306 (158h22m24.819221757s) [noise=true , address=145.79.227.27:4691 , piece_id=AN2YAQFX4APSPXZ37L3THXZDLOU3NZJXZX3GFPQ6GOP2XYMYHJMA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.297086917 (158h22m24.848557379s) [noise=true , address=45.91.116.10:4083 , piece_id=A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.396803830 (158h22m24.948274281s) [noise=true , address=77.90.31.12:4592 , piece_id=DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:51.578031970 (158h22m50.129502431s) [noise=true , address=145.79.227.8:4329 , piece_id=C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:51.578058921 (158h22m50.129529382s) [seg_pos={0 0}, upload=139438]: Segment reached optimal threshold of 13 pieces. Nov 15 22:26:51.578335898 (158h22m50.129806369s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Nov 15 22:26:51.578366757 (158h22m50.129837218s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.578860117 (158h22m50.130330588s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection Nov 15 22:26:51.578892328 (158h22m50.130362789s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.579086789 (158h22m50.13055726s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection Nov 15 22:26:51.579099853 (158h22m50.130570314s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.579150771 (158h22m50.130621232s) [seg_pos={0 0}, upload=139438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 22:26:51.579271630 (158h22m50.130742101s) [upload=139438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 22:26:51.579279916 (158h22m50.130750377s) [upload=139438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 22:26:51.887479665 (158h22m50.438950126s) [upload=139438]: Done uploading segments... Nov 15 22:26:51.887500194 (158h22m50.438970655s) [upload=139438]: Upload finished. err: <nil> Duration: 5.997567448s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.081321965s trace: https://jaeger.dev.storj.io/trace/640930c436a3dd40 FAIL (2025-11-15 22:32:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA) context deadline exceeded; piecestore: (Node ID: 12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, Piece ID: AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA) context deadline exceeded; piecestore: (Node ID: 1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, Piece ID: TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA) context deadline exceeded; piecestore: (Node ID: 12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, Piece ID: DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A) context deadline exceeded ======LOGS====== Nov 15 22:26:23.884265358 (158h22m22.43573582s) [upload=139438]: Got next segment. Inline: false Nov 15 22:26:23.884310675 (158h22m22.435781137s) [upload=139438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 22:26:23.884317198 (158h22m22.435787649s) [upload=139438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 22:26:24.145294674 (158h22m22.696765135s) [seg_pos={0 0}, upload=139438]: Begin upload segment... Nov 15 22:26:24.145353216 (158h22m22.696823667s) [seg_pos={0 0}, upload=139438]: Done begin upload segment. Nov 15 22:26:24.145389284 (158h22m22.696859745s) [noise=true , address=145.79.227.28:4727 , piece_id=JTVEZRINCPKC7U532YI62SEXI25AXEE6ZBXEXWGTI4IWVEPTVWSQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.145586640 (158h22m22.697057091s) [noise=true , address=145.79.227.21:4473 , piece_id=5WUFKUK7GAWMRPMKCGJD3QAXWLE2AH4SP2WN653JYGWZX5OWWGBA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146324976 (158h22m22.697795437s) [noise=true , address=145.79.227.11:4731 , piece_id=2BUZ5Q4KA4MG6EBYHXRICRVAE6MBL6QXMJSIZFV66RBCJ7L6MDKQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146351477 (158h22m22.697821938s) [noise=true , address=45.91.116.11:4117 , piece_id=SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146783870 (158h22m22.698254331s) [noise=true , address=145.79.227.18:4005 , piece_id=AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146833304 (158h22m22.698303765s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147216514 (158h22m22.698686975s) [noise=true , address=145.79.227.10:4402 , piece_id=HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147263273 (158h22m22.698733734s) [noise=true , address=45.91.116.14:4190 , piece_id=X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147675318 (158h22m22.699145799s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147754558 (158h22m22.699225019s) [noise=true , address=145.79.227.20:4439 , piece_id=TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148345744 (158h22m22.699816205s) [noise=true , address=45.91.116.10:4083 , piece_id=A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148407592 (158h22m22.699878053s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148798777 (158h22m22.700269228s) [noise=true , address=145.79.227.31:4799 , piece_id=LT3JYVZJ2A3I4VPLCEYONSNIPRBB4E2XSODHWL572IB3NFCM26JA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148858250 (158h22m22.700328711s) [noise=true , address=145.79.227.27:4691 , piece_id=AN2YAQFX4APSPXZ37L3THXZDLOU3NZJXZX3GFPQ6GOP2XYMYHJMA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.149166006 (158h22m22.700636467s) [noise=true , address=145.79.227.8:4329 , piece_id=C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.149373912 (158h22m22.700844373s) [noise=true , address=77.90.31.12:4592 , piece_id=DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:25.877223267 (158h22m24.428693728s) [upload=139438]: Next returned nil segment Nov 15 22:26:25.877245679 (158h22m24.42871614s) [upload=139438]: Waiting for error group managing segments... Nov 15 22:26:26.154633386 (158h22m24.706103847s) [noise=true , address=145.79.227.31:4799 , piece_id=LT3JYVZJ2A3I4VPLCEYONSNIPRBB4E2XSODHWL572IB3NFCM26JA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.172371583 (158h22m24.723842044s) [noise=true , address=145.79.227.28:4727 , piece_id=JTVEZRINCPKC7U532YI62SEXI25AXEE6ZBXEXWGTI4IWVEPTVWSQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.173052429 (158h22m24.72452289s) [noise=true , address=145.79.227.20:4439 , piece_id=TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.174665170 (158h22m24.726135621s) [noise=true , address=145.79.227.10:4402 , piece_id=HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.175717564 (158h22m24.727188015s) [noise=true , address=145.79.227.11:4731 , piece_id=2BUZ5Q4KA4MG6EBYHXRICRVAE6MBL6QXMJSIZFV66RBCJ7L6MDKQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.203023491 (158h22m24.754493952s) [noise=true , address=45.91.116.14:4190 , piece_id=X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.211675806 (158h22m24.763146267s) [noise=true , address=145.79.227.21:4473 , piece_id=5WUFKUK7GAWMRPMKCGJD3QAXWLE2AH4SP2WN653JYGWZX5OWWGBA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.241230605 (158h22m24.792701066s) [noise=true , address=145.79.227.18:4005 , piece_id=AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.265904560 (158h22m24.817375011s) [noise=true , address=45.91.116.11:4117 , piece_id=SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.267751306 (158h22m24.819221757s) [noise=true , address=145.79.227.27:4691 , piece_id=AN2YAQFX4APSPXZ37L3THXZDLOU3NZJXZX3GFPQ6GOP2XYMYHJMA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.297086917 (158h22m24.848557379s) [noise=true , address=45.91.116.10:4083 , piece_id=A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.396803830 (158h22m24.948274281s) [noise=true , address=77.90.31.12:4592 , piece_id=DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:51.578031970 (158h22m50.129502431s) [noise=true , address=145.79.227.8:4329 , piece_id=C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:51.578058921 (158h22m50.129529382s) [seg_pos={0 0}, upload=139438]: Segment reached optimal threshold of 13 pieces. Nov 15 22:26:51.578335898 (158h22m50.129806369s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Nov 15 22:26:51.578366757 (158h22m50.129837218s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.578860117 (158h22m50.130330588s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection Nov 15 22:26:51.578892328 (158h22m50.130362789s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.579086789 (158h22m50.13055726s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection Nov 15 22:26:51.579099853 (158h22m50.130570314s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.579150771 (158h22m50.130621232s) [seg_pos={0 0}, upload=139438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 22:26:51.579271630 (158h22m50.130742101s) [upload=139438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 22:26:51.579279916 (158h22m50.130750377s) [upload=139438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 22:26:51.887479665 (158h22m50.438950126s) [upload=139438]: Done uploading segments... Nov 15 22:26:51.887500194 (158h22m50.438970655s) [upload=139438]: Upload finished. err: <nil> Duration: 6.081353275s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.112684905s trace: https://jaeger.dev.storj.io/trace/640930c436a3dd40 FAIL (2025-11-15 22:31:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA) context deadline exceeded; piecestore: (Node ID: 12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, Piece ID: AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA) context deadline exceeded; piecestore: (Node ID: 1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, Piece ID: TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA) context deadline exceeded; piecestore: (Node ID: 12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, Piece ID: DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A) context deadline exceeded ======LOGS====== Nov 15 22:26:23.884265358 (158h22m22.43573582s) [upload=139438]: Got next segment. Inline: false Nov 15 22:26:23.884310675 (158h22m22.435781137s) [upload=139438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 22:26:23.884317198 (158h22m22.435787649s) [upload=139438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 22:26:24.145294674 (158h22m22.696765135s) [seg_pos={0 0}, upload=139438]: Begin upload segment... Nov 15 22:26:24.145353216 (158h22m22.696823667s) [seg_pos={0 0}, upload=139438]: Done begin upload segment. Nov 15 22:26:24.145389284 (158h22m22.696859745s) [noise=true , address=145.79.227.28:4727 , piece_id=JTVEZRINCPKC7U532YI62SEXI25AXEE6ZBXEXWGTI4IWVEPTVWSQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.145586640 (158h22m22.697057091s) [noise=true , address=145.79.227.21:4473 , piece_id=5WUFKUK7GAWMRPMKCGJD3QAXWLE2AH4SP2WN653JYGWZX5OWWGBA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146324976 (158h22m22.697795437s) [noise=true , address=145.79.227.11:4731 , piece_id=2BUZ5Q4KA4MG6EBYHXRICRVAE6MBL6QXMJSIZFV66RBCJ7L6MDKQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146351477 (158h22m22.697821938s) [noise=true , address=45.91.116.11:4117 , piece_id=SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146783870 (158h22m22.698254331s) [noise=true , address=145.79.227.18:4005 , piece_id=AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146833304 (158h22m22.698303765s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147216514 (158h22m22.698686975s) [noise=true , address=145.79.227.10:4402 , piece_id=HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147263273 (158h22m22.698733734s) [noise=true , address=45.91.116.14:4190 , piece_id=X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147675318 (158h22m22.699145799s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147754558 (158h22m22.699225019s) [noise=true , address=145.79.227.20:4439 , piece_id=TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148345744 (158h22m22.699816205s) [noise=true , address=45.91.116.10:4083 , piece_id=A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148407592 (158h22m22.699878053s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148798777 (158h22m22.700269228s) [noise=true , address=145.79.227.31:4799 , piece_id=LT3JYVZJ2A3I4VPLCEYONSNIPRBB4E2XSODHWL572IB3NFCM26JA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148858250 (158h22m22.700328711s) [noise=true , address=145.79.227.27:4691 , piece_id=AN2YAQFX4APSPXZ37L3THXZDLOU3NZJXZX3GFPQ6GOP2XYMYHJMA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.149166006 (158h22m22.700636467s) [noise=true , address=145.79.227.8:4329 , piece_id=C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.149373912 (158h22m22.700844373s) [noise=true , address=77.90.31.12:4592 , piece_id=DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:25.877223267 (158h22m24.428693728s) [upload=139438]: Next returned nil segment Nov 15 22:26:25.877245679 (158h22m24.42871614s) [upload=139438]: Waiting for error group managing segments... Nov 15 22:26:26.154633386 (158h22m24.706103847s) [noise=true , address=145.79.227.31:4799 , piece_id=LT3JYVZJ2A3I4VPLCEYONSNIPRBB4E2XSODHWL572IB3NFCM26JA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.172371583 (158h22m24.723842044s) [noise=true , address=145.79.227.28:4727 , piece_id=JTVEZRINCPKC7U532YI62SEXI25AXEE6ZBXEXWGTI4IWVEPTVWSQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.173052429 (158h22m24.72452289s) [noise=true , address=145.79.227.20:4439 , piece_id=TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.174665170 (158h22m24.726135621s) [noise=true , address=145.79.227.10:4402 , piece_id=HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.175717564 (158h22m24.727188015s) [noise=true , address=145.79.227.11:4731 , piece_id=2BUZ5Q4KA4MG6EBYHXRICRVAE6MBL6QXMJSIZFV66RBCJ7L6MDKQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.203023491 (158h22m24.754493952s) [noise=true , address=45.91.116.14:4190 , piece_id=X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.211675806 (158h22m24.763146267s) [noise=true , address=145.79.227.21:4473 , piece_id=5WUFKUK7GAWMRPMKCGJD3QAXWLE2AH4SP2WN653JYGWZX5OWWGBA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.241230605 (158h22m24.792701066s) [noise=true , address=145.79.227.18:4005 , piece_id=AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.265904560 (158h22m24.817375011s) [noise=true , address=45.91.116.11:4117 , piece_id=SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.267751306 (158h22m24.819221757s) [noise=true , address=145.79.227.27:4691 , piece_id=AN2YAQFX4APSPXZ37L3THXZDLOU3NZJXZX3GFPQ6GOP2XYMYHJMA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.297086917 (158h22m24.848557379s) [noise=true , address=45.91.116.10:4083 , piece_id=A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.396803830 (158h22m24.948274281s) [noise=true , address=77.90.31.12:4592 , piece_id=DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:51.578031970 (158h22m50.129502431s) [noise=true , address=145.79.227.8:4329 , piece_id=C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:51.578058921 (158h22m50.129529382s) [seg_pos={0 0}, upload=139438]: Segment reached optimal threshold of 13 pieces. Nov 15 22:26:51.578335898 (158h22m50.129806369s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Nov 15 22:26:51.578366757 (158h22m50.129837218s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.578860117 (158h22m50.130330588s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection Nov 15 22:26:51.578892328 (158h22m50.130362789s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.579086789 (158h22m50.13055726s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection Nov 15 22:26:51.579099853 (158h22m50.130570314s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.579150771 (158h22m50.130621232s) [seg_pos={0 0}, upload=139438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 22:26:51.579271630 (158h22m50.130742101s) [upload=139438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 22:26:51.579279916 (158h22m50.130750377s) [upload=139438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 22:26:51.887479665 (158h22m50.438950126s) [upload=139438]: Done uploading segments... Nov 15 22:26:51.887500194 (158h22m50.438970655s) [upload=139438]: Upload finished. err: <nil> Duration: 6.112712648s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.017209745s trace: https://jaeger.dev.storj.io/trace/640930c436a3dd40 FAIL (2025-11-15 22:30:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA) context deadline exceeded; piecestore: (Node ID: 12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, Piece ID: AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA) context deadline exceeded; piecestore: (Node ID: 1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, Piece ID: TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA) context deadline exceeded; piecestore: (Node ID: 12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, Piece ID: DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A) context deadline exceeded ======LOGS====== Nov 15 22:26:23.884265358 (158h22m22.43573582s) [upload=139438]: Got next segment. Inline: false Nov 15 22:26:23.884310675 (158h22m22.435781137s) [upload=139438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 22:26:23.884317198 (158h22m22.435787649s) [upload=139438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 22:26:24.145294674 (158h22m22.696765135s) [seg_pos={0 0}, upload=139438]: Begin upload segment... Nov 15 22:26:24.145353216 (158h22m22.696823667s) [seg_pos={0 0}, upload=139438]: Done begin upload segment. Nov 15 22:26:24.145389284 (158h22m22.696859745s) [noise=true , address=145.79.227.28:4727 , piece_id=JTVEZRINCPKC7U532YI62SEXI25AXEE6ZBXEXWGTI4IWVEPTVWSQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.145586640 (158h22m22.697057091s) [noise=true , address=145.79.227.21:4473 , piece_id=5WUFKUK7GAWMRPMKCGJD3QAXWLE2AH4SP2WN653JYGWZX5OWWGBA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146324976 (158h22m22.697795437s) [noise=true , address=145.79.227.11:4731 , piece_id=2BUZ5Q4KA4MG6EBYHXRICRVAE6MBL6QXMJSIZFV66RBCJ7L6MDKQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146351477 (158h22m22.697821938s) [noise=true , address=45.91.116.11:4117 , piece_id=SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146783870 (158h22m22.698254331s) [noise=true , address=145.79.227.18:4005 , piece_id=AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146833304 (158h22m22.698303765s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147216514 (158h22m22.698686975s) [noise=true , address=145.79.227.10:4402 , piece_id=HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147263273 (158h22m22.698733734s) [noise=true , address=45.91.116.14:4190 , piece_id=X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147675318 (158h22m22.699145799s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147754558 (158h22m22.699225019s) [noise=true , address=145.79.227.20:4439 , piece_id=TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148345744 (158h22m22.699816205s) [noise=true , address=45.91.116.10:4083 , piece_id=A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148407592 (158h22m22.699878053s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148798777 (158h22m22.700269228s) [noise=true , address=145.79.227.31:4799 , piece_id=LT3JYVZJ2A3I4VPLCEYONSNIPRBB4E2XSODHWL572IB3NFCM26JA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148858250 (158h22m22.700328711s) [noise=true , address=145.79.227.27:4691 , piece_id=AN2YAQFX4APSPXZ37L3THXZDLOU3NZJXZX3GFPQ6GOP2XYMYHJMA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.149166006 (158h22m22.700636467s) [noise=true , address=145.79.227.8:4329 , piece_id=C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.149373912 (158h22m22.700844373s) [noise=true , address=77.90.31.12:4592 , piece_id=DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:25.877223267 (158h22m24.428693728s) [upload=139438]: Next returned nil segment Nov 15 22:26:25.877245679 (158h22m24.42871614s) [upload=139438]: Waiting for error group managing segments... Nov 15 22:26:26.154633386 (158h22m24.706103847s) [noise=true , address=145.79.227.31:4799 , piece_id=LT3JYVZJ2A3I4VPLCEYONSNIPRBB4E2XSODHWL572IB3NFCM26JA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.172371583 (158h22m24.723842044s) [noise=true , address=145.79.227.28:4727 , piece_id=JTVEZRINCPKC7U532YI62SEXI25AXEE6ZBXEXWGTI4IWVEPTVWSQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.173052429 (158h22m24.72452289s) [noise=true , address=145.79.227.20:4439 , piece_id=TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.174665170 (158h22m24.726135621s) [noise=true , address=145.79.227.10:4402 , piece_id=HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.175717564 (158h22m24.727188015s) [noise=true , address=145.79.227.11:4731 , piece_id=2BUZ5Q4KA4MG6EBYHXRICRVAE6MBL6QXMJSIZFV66RBCJ7L6MDKQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.203023491 (158h22m24.754493952s) [noise=true , address=45.91.116.14:4190 , piece_id=X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.211675806 (158h22m24.763146267s) [noise=true , address=145.79.227.21:4473 , piece_id=5WUFKUK7GAWMRPMKCGJD3QAXWLE2AH4SP2WN653JYGWZX5OWWGBA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.241230605 (158h22m24.792701066s) [noise=true , address=145.79.227.18:4005 , piece_id=AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.265904560 (158h22m24.817375011s) [noise=true , address=45.91.116.11:4117 , piece_id=SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.267751306 (158h22m24.819221757s) [noise=true , address=145.79.227.27:4691 , piece_id=AN2YAQFX4APSPXZ37L3THXZDLOU3NZJXZX3GFPQ6GOP2XYMYHJMA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.297086917 (158h22m24.848557379s) [noise=true , address=45.91.116.10:4083 , piece_id=A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.396803830 (158h22m24.948274281s) [noise=true , address=77.90.31.12:4592 , piece_id=DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:51.578031970 (158h22m50.129502431s) [noise=true , address=145.79.227.8:4329 , piece_id=C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:51.578058921 (158h22m50.129529382s) [seg_pos={0 0}, upload=139438]: Segment reached optimal threshold of 13 pieces. Nov 15 22:26:51.578335898 (158h22m50.129806369s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Nov 15 22:26:51.578366757 (158h22m50.129837218s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.578860117 (158h22m50.130330588s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection Nov 15 22:26:51.578892328 (158h22m50.130362789s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.579086789 (158h22m50.13055726s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection Nov 15 22:26:51.579099853 (158h22m50.130570314s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.579150771 (158h22m50.130621232s) [seg_pos={0 0}, upload=139438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 22:26:51.579271630 (158h22m50.130742101s) [upload=139438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 22:26:51.579279916 (158h22m50.130750377s) [upload=139438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 22:26:51.887479665 (158h22m50.438950126s) [upload=139438]: Done uploading segments... Nov 15 22:26:51.887500194 (158h22m50.438970655s) [upload=139438]: Upload finished. err: <nil> Duration: 6.017231737s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.177918941s trace: https://jaeger.dev.storj.io/trace/640930c436a3dd40 FAIL (2025-11-15 22:29:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA) context deadline exceeded; piecestore: (Node ID: 12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, Piece ID: AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA) context deadline exceeded; piecestore: (Node ID: 1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, Piece ID: TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA) context deadline exceeded; piecestore: (Node ID: 12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, Piece ID: DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A) context deadline exceeded ======LOGS====== Nov 15 22:26:23.884265358 (158h22m22.43573582s) [upload=139438]: Got next segment. Inline: false Nov 15 22:26:23.884310675 (158h22m22.435781137s) [upload=139438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 22:26:23.884317198 (158h22m22.435787649s) [upload=139438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 22:26:24.145294674 (158h22m22.696765135s) [seg_pos={0 0}, upload=139438]: Begin upload segment... Nov 15 22:26:24.145353216 (158h22m22.696823667s) [seg_pos={0 0}, upload=139438]: Done begin upload segment. Nov 15 22:26:24.145389284 (158h22m22.696859745s) [noise=true , address=145.79.227.28:4727 , piece_id=JTVEZRINCPKC7U532YI62SEXI25AXEE6ZBXEXWGTI4IWVEPTVWSQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.145586640 (158h22m22.697057091s) [noise=true , address=145.79.227.21:4473 , piece_id=5WUFKUK7GAWMRPMKCGJD3QAXWLE2AH4SP2WN653JYGWZX5OWWGBA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146324976 (158h22m22.697795437s) [noise=true , address=145.79.227.11:4731 , piece_id=2BUZ5Q4KA4MG6EBYHXRICRVAE6MBL6QXMJSIZFV66RBCJ7L6MDKQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146351477 (158h22m22.697821938s) [noise=true , address=45.91.116.11:4117 , piece_id=SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146783870 (158h22m22.698254331s) [noise=true , address=145.79.227.18:4005 , piece_id=AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146833304 (158h22m22.698303765s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147216514 (158h22m22.698686975s) [noise=true , address=145.79.227.10:4402 , piece_id=HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147263273 (158h22m22.698733734s) [noise=true , address=45.91.116.14:4190 , piece_id=X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147675318 (158h22m22.699145799s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147754558 (158h22m22.699225019s) [noise=true , address=145.79.227.20:4439 , piece_id=TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148345744 (158h22m22.699816205s) [noise=true , address=45.91.116.10:4083 , piece_id=A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148407592 (158h22m22.699878053s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148798777 (158h22m22.700269228s) [noise=true , address=145.79.227.31:4799 , piece_id=LT3JYVZJ2A3I4VPLCEYONSNIPRBB4E2XSODHWL572IB3NFCM26JA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148858250 (158h22m22.700328711s) [noise=true , address=145.79.227.27:4691 , piece_id=AN2YAQFX4APSPXZ37L3THXZDLOU3NZJXZX3GFPQ6GOP2XYMYHJMA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.149166006 (158h22m22.700636467s) [noise=true , address=145.79.227.8:4329 , piece_id=C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.149373912 (158h22m22.700844373s) [noise=true , address=77.90.31.12:4592 , piece_id=DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:25.877223267 (158h22m24.428693728s) [upload=139438]: Next returned nil segment Nov 15 22:26:25.877245679 (158h22m24.42871614s) [upload=139438]: Waiting for error group managing segments... Nov 15 22:26:26.154633386 (158h22m24.706103847s) [noise=true , address=145.79.227.31:4799 , piece_id=LT3JYVZJ2A3I4VPLCEYONSNIPRBB4E2XSODHWL572IB3NFCM26JA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.172371583 (158h22m24.723842044s) [noise=true , address=145.79.227.28:4727 , piece_id=JTVEZRINCPKC7U532YI62SEXI25AXEE6ZBXEXWGTI4IWVEPTVWSQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.173052429 (158h22m24.72452289s) [noise=true , address=145.79.227.20:4439 , piece_id=TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.174665170 (158h22m24.726135621s) [noise=true , address=145.79.227.10:4402 , piece_id=HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.175717564 (158h22m24.727188015s) [noise=true , address=145.79.227.11:4731 , piece_id=2BUZ5Q4KA4MG6EBYHXRICRVAE6MBL6QXMJSIZFV66RBCJ7L6MDKQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.203023491 (158h22m24.754493952s) [noise=true , address=45.91.116.14:4190 , piece_id=X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.211675806 (158h22m24.763146267s) [noise=true , address=145.79.227.21:4473 , piece_id=5WUFKUK7GAWMRPMKCGJD3QAXWLE2AH4SP2WN653JYGWZX5OWWGBA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.241230605 (158h22m24.792701066s) [noise=true , address=145.79.227.18:4005 , piece_id=AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.265904560 (158h22m24.817375011s) [noise=true , address=45.91.116.11:4117 , piece_id=SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.267751306 (158h22m24.819221757s) [noise=true , address=145.79.227.27:4691 , piece_id=AN2YAQFX4APSPXZ37L3THXZDLOU3NZJXZX3GFPQ6GOP2XYMYHJMA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.297086917 (158h22m24.848557379s) [noise=true , address=45.91.116.10:4083 , piece_id=A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.396803830 (158h22m24.948274281s) [noise=true , address=77.90.31.12:4592 , piece_id=DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:51.578031970 (158h22m50.129502431s) [noise=true , address=145.79.227.8:4329 , piece_id=C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:51.578058921 (158h22m50.129529382s) [seg_pos={0 0}, upload=139438]: Segment reached optimal threshold of 13 pieces. Nov 15 22:26:51.578335898 (158h22m50.129806369s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Nov 15 22:26:51.578366757 (158h22m50.129837218s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.578860117 (158h22m50.130330588s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection Nov 15 22:26:51.578892328 (158h22m50.130362789s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.579086789 (158h22m50.13055726s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection Nov 15 22:26:51.579099853 (158h22m50.130570314s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.579150771 (158h22m50.130621232s) [seg_pos={0 0}, upload=139438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 22:26:51.579271630 (158h22m50.130742101s) [upload=139438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 22:26:51.579279916 (158h22m50.130750377s) [upload=139438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 22:26:51.887479665 (158h22m50.438950126s) [upload=139438]: Done uploading segments... Nov 15 22:26:51.887500194 (158h22m50.438970655s) [upload=139438]: Upload finished. err: <nil> Duration: 6.177948849s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.977515911s trace: https://jaeger.dev.storj.io/trace/640930c436a3dd40 FAIL (2025-11-15 22:29:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA) context deadline exceeded; piecestore: (Node ID: 12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, Piece ID: AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA) context deadline exceeded; piecestore: (Node ID: 1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, Piece ID: TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA) context deadline exceeded; piecestore: (Node ID: 12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, Piece ID: DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A) context deadline exceeded ======LOGS====== Nov 15 22:26:23.884265358 (158h22m22.43573582s) [upload=139438]: Got next segment. Inline: false Nov 15 22:26:23.884310675 (158h22m22.435781137s) [upload=139438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 22:26:23.884317198 (158h22m22.435787649s) [upload=139438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 22:26:24.145294674 (158h22m22.696765135s) [seg_pos={0 0}, upload=139438]: Begin upload segment... Nov 15 22:26:24.145353216 (158h22m22.696823667s) [seg_pos={0 0}, upload=139438]: Done begin upload segment. Nov 15 22:26:24.145389284 (158h22m22.696859745s) [noise=true , address=145.79.227.28:4727 , piece_id=JTVEZRINCPKC7U532YI62SEXI25AXEE6ZBXEXWGTI4IWVEPTVWSQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.145586640 (158h22m22.697057091s) [noise=true , address=145.79.227.21:4473 , piece_id=5WUFKUK7GAWMRPMKCGJD3QAXWLE2AH4SP2WN653JYGWZX5OWWGBA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146324976 (158h22m22.697795437s) [noise=true , address=145.79.227.11:4731 , piece_id=2BUZ5Q4KA4MG6EBYHXRICRVAE6MBL6QXMJSIZFV66RBCJ7L6MDKQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146351477 (158h22m22.697821938s) [noise=true , address=45.91.116.11:4117 , piece_id=SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146783870 (158h22m22.698254331s) [noise=true , address=145.79.227.18:4005 , piece_id=AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146833304 (158h22m22.698303765s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147216514 (158h22m22.698686975s) [noise=true , address=145.79.227.10:4402 , piece_id=HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147263273 (158h22m22.698733734s) [noise=true , address=45.91.116.14:4190 , piece_id=X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147675318 (158h22m22.699145799s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147754558 (158h22m22.699225019s) [noise=true , address=145.79.227.20:4439 , piece_id=TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148345744 (158h22m22.699816205s) [noise=true , address=45.91.116.10:4083 , piece_id=A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148407592 (158h22m22.699878053s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148798777 (158h22m22.700269228s) [noise=true , address=145.79.227.31:4799 , piece_id=LT3JYVZJ2A3I4VPLCEYONSNIPRBB4E2XSODHWL572IB3NFCM26JA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148858250 (158h22m22.700328711s) [noise=true , address=145.79.227.27:4691 , piece_id=AN2YAQFX4APSPXZ37L3THXZDLOU3NZJXZX3GFPQ6GOP2XYMYHJMA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.149166006 (158h22m22.700636467s) [noise=true , address=145.79.227.8:4329 , piece_id=C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.149373912 (158h22m22.700844373s) [noise=true , address=77.90.31.12:4592 , piece_id=DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:25.877223267 (158h22m24.428693728s) [upload=139438]: Next returned nil segment Nov 15 22:26:25.877245679 (158h22m24.42871614s) [upload=139438]: Waiting for error group managing segments... Nov 15 22:26:26.154633386 (158h22m24.706103847s) [noise=true , address=145.79.227.31:4799 , piece_id=LT3JYVZJ2A3I4VPLCEYONSNIPRBB4E2XSODHWL572IB3NFCM26JA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.172371583 (158h22m24.723842044s) [noise=true , address=145.79.227.28:4727 , piece_id=JTVEZRINCPKC7U532YI62SEXI25AXEE6ZBXEXWGTI4IWVEPTVWSQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.173052429 (158h22m24.72452289s) [noise=true , address=145.79.227.20:4439 , piece_id=TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.174665170 (158h22m24.726135621s) [noise=true , address=145.79.227.10:4402 , piece_id=HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.175717564 (158h22m24.727188015s) [noise=true , address=145.79.227.11:4731 , piece_id=2BUZ5Q4KA4MG6EBYHXRICRVAE6MBL6QXMJSIZFV66RBCJ7L6MDKQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.203023491 (158h22m24.754493952s) [noise=true , address=45.91.116.14:4190 , piece_id=X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.211675806 (158h22m24.763146267s) [noise=true , address=145.79.227.21:4473 , piece_id=5WUFKUK7GAWMRPMKCGJD3QAXWLE2AH4SP2WN653JYGWZX5OWWGBA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.241230605 (158h22m24.792701066s) [noise=true , address=145.79.227.18:4005 , piece_id=AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.265904560 (158h22m24.817375011s) [noise=true , address=45.91.116.11:4117 , piece_id=SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.267751306 (158h22m24.819221757s) [noise=true , address=145.79.227.27:4691 , piece_id=AN2YAQFX4APSPXZ37L3THXZDLOU3NZJXZX3GFPQ6GOP2XYMYHJMA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.297086917 (158h22m24.848557379s) [noise=true , address=45.91.116.10:4083 , piece_id=A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.396803830 (158h22m24.948274281s) [noise=true , address=77.90.31.12:4592 , piece_id=DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:51.578031970 (158h22m50.129502431s) [noise=true , address=145.79.227.8:4329 , piece_id=C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:51.578058921 (158h22m50.129529382s) [seg_pos={0 0}, upload=139438]: Segment reached optimal threshold of 13 pieces. Nov 15 22:26:51.578335898 (158h22m50.129806369s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Nov 15 22:26:51.578366757 (158h22m50.129837218s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.578860117 (158h22m50.130330588s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection Nov 15 22:26:51.578892328 (158h22m50.130362789s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.579086789 (158h22m50.13055726s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection Nov 15 22:26:51.579099853 (158h22m50.130570314s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.579150771 (158h22m50.130621232s) [seg_pos={0 0}, upload=139438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 22:26:51.579271630 (158h22m50.130742101s) [upload=139438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 22:26:51.579279916 (158h22m50.130750377s) [upload=139438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 22:26:51.887479665 (158h22m50.438950126s) [upload=139438]: Done uploading segments... Nov 15 22:26:51.887500194 (158h22m50.438970655s) [upload=139438]: Upload finished. err: <nil> Duration: 5.977544274s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.06127415s trace: https://jaeger.dev.storj.io/trace/640930c436a3dd40 FAIL (2025-11-15 22:28:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA) context deadline exceeded; piecestore: (Node ID: 12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, Piece ID: AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA) context deadline exceeded; piecestore: (Node ID: 1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, Piece ID: TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA) context deadline exceeded; piecestore: (Node ID: 12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, Piece ID: DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A) context deadline exceeded ======LOGS====== Nov 15 22:26:23.884265358 (158h22m22.43573582s) [upload=139438]: Got next segment. Inline: false Nov 15 22:26:23.884310675 (158h22m22.435781137s) [upload=139438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 22:26:23.884317198 (158h22m22.435787649s) [upload=139438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 22:26:24.145294674 (158h22m22.696765135s) [seg_pos={0 0}, upload=139438]: Begin upload segment... Nov 15 22:26:24.145353216 (158h22m22.696823667s) [seg_pos={0 0}, upload=139438]: Done begin upload segment. Nov 15 22:26:24.145389284 (158h22m22.696859745s) [noise=true , address=145.79.227.28:4727 , piece_id=JTVEZRINCPKC7U532YI62SEXI25AXEE6ZBXEXWGTI4IWVEPTVWSQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.145586640 (158h22m22.697057091s) [noise=true , address=145.79.227.21:4473 , piece_id=5WUFKUK7GAWMRPMKCGJD3QAXWLE2AH4SP2WN653JYGWZX5OWWGBA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146324976 (158h22m22.697795437s) [noise=true , address=145.79.227.11:4731 , piece_id=2BUZ5Q4KA4MG6EBYHXRICRVAE6MBL6QXMJSIZFV66RBCJ7L6MDKQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146351477 (158h22m22.697821938s) [noise=true , address=45.91.116.11:4117 , piece_id=SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146783870 (158h22m22.698254331s) [noise=true , address=145.79.227.18:4005 , piece_id=AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146833304 (158h22m22.698303765s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147216514 (158h22m22.698686975s) [noise=true , address=145.79.227.10:4402 , piece_id=HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147263273 (158h22m22.698733734s) [noise=true , address=45.91.116.14:4190 , piece_id=X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147675318 (158h22m22.699145799s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147754558 (158h22m22.699225019s) [noise=true , address=145.79.227.20:4439 , piece_id=TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148345744 (158h22m22.699816205s) [noise=true , address=45.91.116.10:4083 , piece_id=A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148407592 (158h22m22.699878053s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148798777 (158h22m22.700269228s) [noise=true , address=145.79.227.31:4799 , piece_id=LT3JYVZJ2A3I4VPLCEYONSNIPRBB4E2XSODHWL572IB3NFCM26JA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148858250 (158h22m22.700328711s) [noise=true , address=145.79.227.27:4691 , piece_id=AN2YAQFX4APSPXZ37L3THXZDLOU3NZJXZX3GFPQ6GOP2XYMYHJMA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.149166006 (158h22m22.700636467s) [noise=true , address=145.79.227.8:4329 , piece_id=C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.149373912 (158h22m22.700844373s) [noise=true , address=77.90.31.12:4592 , piece_id=DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:25.877223267 (158h22m24.428693728s) [upload=139438]: Next returned nil segment Nov 15 22:26:25.877245679 (158h22m24.42871614s) [upload=139438]: Waiting for error group managing segments... Nov 15 22:26:26.154633386 (158h22m24.706103847s) [noise=true , address=145.79.227.31:4799 , piece_id=LT3JYVZJ2A3I4VPLCEYONSNIPRBB4E2XSODHWL572IB3NFCM26JA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.172371583 (158h22m24.723842044s) [noise=true , address=145.79.227.28:4727 , piece_id=JTVEZRINCPKC7U532YI62SEXI25AXEE6ZBXEXWGTI4IWVEPTVWSQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.173052429 (158h22m24.72452289s) [noise=true , address=145.79.227.20:4439 , piece_id=TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.174665170 (158h22m24.726135621s) [noise=true , address=145.79.227.10:4402 , piece_id=HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.175717564 (158h22m24.727188015s) [noise=true , address=145.79.227.11:4731 , piece_id=2BUZ5Q4KA4MG6EBYHXRICRVAE6MBL6QXMJSIZFV66RBCJ7L6MDKQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.203023491 (158h22m24.754493952s) [noise=true , address=45.91.116.14:4190 , piece_id=X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.211675806 (158h22m24.763146267s) [noise=true , address=145.79.227.21:4473 , piece_id=5WUFKUK7GAWMRPMKCGJD3QAXWLE2AH4SP2WN653JYGWZX5OWWGBA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.241230605 (158h22m24.792701066s) [noise=true , address=145.79.227.18:4005 , piece_id=AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.265904560 (158h22m24.817375011s) [noise=true , address=45.91.116.11:4117 , piece_id=SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.267751306 (158h22m24.819221757s) [noise=true , address=145.79.227.27:4691 , piece_id=AN2YAQFX4APSPXZ37L3THXZDLOU3NZJXZX3GFPQ6GOP2XYMYHJMA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.297086917 (158h22m24.848557379s) [noise=true , address=45.91.116.10:4083 , piece_id=A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.396803830 (158h22m24.948274281s) [noise=true , address=77.90.31.12:4592 , piece_id=DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:51.578031970 (158h22m50.129502431s) [noise=true , address=145.79.227.8:4329 , piece_id=C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:51.578058921 (158h22m50.129529382s) [seg_pos={0 0}, upload=139438]: Segment reached optimal threshold of 13 pieces. Nov 15 22:26:51.578335898 (158h22m50.129806369s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Nov 15 22:26:51.578366757 (158h22m50.129837218s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.578860117 (158h22m50.130330588s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection Nov 15 22:26:51.578892328 (158h22m50.130362789s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.579086789 (158h22m50.13055726s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection Nov 15 22:26:51.579099853 (158h22m50.130570314s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.579150771 (158h22m50.130621232s) [seg_pos={0 0}, upload=139438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 22:26:51.579271630 (158h22m50.130742101s) [upload=139438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 22:26:51.579279916 (158h22m50.130750377s) [upload=139438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 22:26:51.887479665 (158h22m50.438950126s) [upload=139438]: Done uploading segments... Nov 15 22:26:51.887500194 (158h22m50.438970655s) [upload=139438]: Upload finished. err: <nil> Duration: 6.061303526s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.075310928s trace: https://jaeger.dev.storj.io/trace/640930c436a3dd40 FAIL (2025-11-15 22:27:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA) context deadline exceeded; piecestore: (Node ID: 12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, Piece ID: AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA) context deadline exceeded; piecestore: (Node ID: 1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, Piece ID: TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA) context deadline exceeded; piecestore: (Node ID: 12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, Piece ID: DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A) context deadline exceeded ======LOGS====== Nov 15 22:26:23.884265358 (158h22m22.43573582s) [upload=139438]: Got next segment. Inline: false Nov 15 22:26:23.884310675 (158h22m22.435781137s) [upload=139438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 22:26:23.884317198 (158h22m22.435787649s) [upload=139438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 22:26:24.145294674 (158h22m22.696765135s) [seg_pos={0 0}, upload=139438]: Begin upload segment... Nov 15 22:26:24.145353216 (158h22m22.696823667s) [seg_pos={0 0}, upload=139438]: Done begin upload segment. Nov 15 22:26:24.145389284 (158h22m22.696859745s) [noise=true , address=145.79.227.28:4727 , piece_id=JTVEZRINCPKC7U532YI62SEXI25AXEE6ZBXEXWGTI4IWVEPTVWSQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.145586640 (158h22m22.697057091s) [noise=true , address=145.79.227.21:4473 , piece_id=5WUFKUK7GAWMRPMKCGJD3QAXWLE2AH4SP2WN653JYGWZX5OWWGBA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146324976 (158h22m22.697795437s) [noise=true , address=145.79.227.11:4731 , piece_id=2BUZ5Q4KA4MG6EBYHXRICRVAE6MBL6QXMJSIZFV66RBCJ7L6MDKQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146351477 (158h22m22.697821938s) [noise=true , address=45.91.116.11:4117 , piece_id=SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146783870 (158h22m22.698254331s) [noise=true , address=145.79.227.18:4005 , piece_id=AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146833304 (158h22m22.698303765s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147216514 (158h22m22.698686975s) [noise=true , address=145.79.227.10:4402 , piece_id=HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147263273 (158h22m22.698733734s) [noise=true , address=45.91.116.14:4190 , piece_id=X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147675318 (158h22m22.699145799s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147754558 (158h22m22.699225019s) [noise=true , address=145.79.227.20:4439 , piece_id=TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148345744 (158h22m22.699816205s) [noise=true , address=45.91.116.10:4083 , piece_id=A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148407592 (158h22m22.699878053s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148798777 (158h22m22.700269228s) [noise=true , address=145.79.227.31:4799 , piece_id=LT3JYVZJ2A3I4VPLCEYONSNIPRBB4E2XSODHWL572IB3NFCM26JA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148858250 (158h22m22.700328711s) [noise=true , address=145.79.227.27:4691 , piece_id=AN2YAQFX4APSPXZ37L3THXZDLOU3NZJXZX3GFPQ6GOP2XYMYHJMA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.149166006 (158h22m22.700636467s) [noise=true , address=145.79.227.8:4329 , piece_id=C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.149373912 (158h22m22.700844373s) [noise=true , address=77.90.31.12:4592 , piece_id=DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:25.877223267 (158h22m24.428693728s) [upload=139438]: Next returned nil segment Nov 15 22:26:25.877245679 (158h22m24.42871614s) [upload=139438]: Waiting for error group managing segments... Nov 15 22:26:26.154633386 (158h22m24.706103847s) [noise=true , address=145.79.227.31:4799 , piece_id=LT3JYVZJ2A3I4VPLCEYONSNIPRBB4E2XSODHWL572IB3NFCM26JA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.172371583 (158h22m24.723842044s) [noise=true , address=145.79.227.28:4727 , piece_id=JTVEZRINCPKC7U532YI62SEXI25AXEE6ZBXEXWGTI4IWVEPTVWSQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.173052429 (158h22m24.72452289s) [noise=true , address=145.79.227.20:4439 , piece_id=TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.174665170 (158h22m24.726135621s) [noise=true , address=145.79.227.10:4402 , piece_id=HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.175717564 (158h22m24.727188015s) [noise=true , address=145.79.227.11:4731 , piece_id=2BUZ5Q4KA4MG6EBYHXRICRVAE6MBL6QXMJSIZFV66RBCJ7L6MDKQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.203023491 (158h22m24.754493952s) [noise=true , address=45.91.116.14:4190 , piece_id=X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.211675806 (158h22m24.763146267s) [noise=true , address=145.79.227.21:4473 , piece_id=5WUFKUK7GAWMRPMKCGJD3QAXWLE2AH4SP2WN653JYGWZX5OWWGBA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.241230605 (158h22m24.792701066s) [noise=true , address=145.79.227.18:4005 , piece_id=AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.265904560 (158h22m24.817375011s) [noise=true , address=45.91.116.11:4117 , piece_id=SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.267751306 (158h22m24.819221757s) [noise=true , address=145.79.227.27:4691 , piece_id=AN2YAQFX4APSPXZ37L3THXZDLOU3NZJXZX3GFPQ6GOP2XYMYHJMA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.297086917 (158h22m24.848557379s) [noise=true , address=45.91.116.10:4083 , piece_id=A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.396803830 (158h22m24.948274281s) [noise=true , address=77.90.31.12:4592 , piece_id=DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:51.578031970 (158h22m50.129502431s) [noise=true , address=145.79.227.8:4329 , piece_id=C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:51.578058921 (158h22m50.129529382s) [seg_pos={0 0}, upload=139438]: Segment reached optimal threshold of 13 pieces. Nov 15 22:26:51.578335898 (158h22m50.129806369s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Nov 15 22:26:51.578366757 (158h22m50.129837218s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.578860117 (158h22m50.130330588s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection Nov 15 22:26:51.578892328 (158h22m50.130362789s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.579086789 (158h22m50.13055726s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection Nov 15 22:26:51.579099853 (158h22m50.130570314s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.579150771 (158h22m50.130621232s) [seg_pos={0 0}, upload=139438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 22:26:51.579271630 (158h22m50.130742101s) [upload=139438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 22:26:51.579279916 (158h22m50.130750377s) [upload=139438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 22:26:51.887479665 (158h22m50.438950126s) [upload=139438]: Done uploading segments... Nov 15 22:26:51.887500194 (158h22m50.438970655s) [upload=139438]: Upload finished. err: <nil> Duration: 6.075346355s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.923986665s trace: https://jaeger.dev.storj.io/trace/640930c436a3dd40 FAIL (2025-11-15 22:27:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA) context deadline exceeded; piecestore: (Node ID: 12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, Piece ID: AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA) context deadline exceeded; piecestore: (Node ID: 1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, Piece ID: TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA) context deadline exceeded; piecestore: (Node ID: 12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, Piece ID: DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A) context deadline exceeded ======LOGS====== Nov 15 22:26:23.884265358 (158h22m22.43573582s) [upload=139438]: Got next segment. Inline: false Nov 15 22:26:23.884310675 (158h22m22.435781137s) [upload=139438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 22:26:23.884317198 (158h22m22.435787649s) [upload=139438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 22:26:24.145294674 (158h22m22.696765135s) [seg_pos={0 0}, upload=139438]: Begin upload segment... Nov 15 22:26:24.145353216 (158h22m22.696823667s) [seg_pos={0 0}, upload=139438]: Done begin upload segment. Nov 15 22:26:24.145389284 (158h22m22.696859745s) [noise=true , address=145.79.227.28:4727 , piece_id=JTVEZRINCPKC7U532YI62SEXI25AXEE6ZBXEXWGTI4IWVEPTVWSQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.145586640 (158h22m22.697057091s) [noise=true , address=145.79.227.21:4473 , piece_id=5WUFKUK7GAWMRPMKCGJD3QAXWLE2AH4SP2WN653JYGWZX5OWWGBA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146324976 (158h22m22.697795437s) [noise=true , address=145.79.227.11:4731 , piece_id=2BUZ5Q4KA4MG6EBYHXRICRVAE6MBL6QXMJSIZFV66RBCJ7L6MDKQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146351477 (158h22m22.697821938s) [noise=true , address=45.91.116.11:4117 , piece_id=SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146783870 (158h22m22.698254331s) [noise=true , address=145.79.227.18:4005 , piece_id=AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146833304 (158h22m22.698303765s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147216514 (158h22m22.698686975s) [noise=true , address=145.79.227.10:4402 , piece_id=HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147263273 (158h22m22.698733734s) [noise=true , address=45.91.116.14:4190 , piece_id=X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147675318 (158h22m22.699145799s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147754558 (158h22m22.699225019s) [noise=true , address=145.79.227.20:4439 , piece_id=TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148345744 (158h22m22.699816205s) [noise=true , address=45.91.116.10:4083 , piece_id=A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148407592 (158h22m22.699878053s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148798777 (158h22m22.700269228s) [noise=true , address=145.79.227.31:4799 , piece_id=LT3JYVZJ2A3I4VPLCEYONSNIPRBB4E2XSODHWL572IB3NFCM26JA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148858250 (158h22m22.700328711s) [noise=true , address=145.79.227.27:4691 , piece_id=AN2YAQFX4APSPXZ37L3THXZDLOU3NZJXZX3GFPQ6GOP2XYMYHJMA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.149166006 (158h22m22.700636467s) [noise=true , address=145.79.227.8:4329 , piece_id=C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.149373912 (158h22m22.700844373s) [noise=true , address=77.90.31.12:4592 , piece_id=DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:25.877223267 (158h22m24.428693728s) [upload=139438]: Next returned nil segment Nov 15 22:26:25.877245679 (158h22m24.42871614s) [upload=139438]: Waiting for error group managing segments... Nov 15 22:26:26.154633386 (158h22m24.706103847s) [noise=true , address=145.79.227.31:4799 , piece_id=LT3JYVZJ2A3I4VPLCEYONSNIPRBB4E2XSODHWL572IB3NFCM26JA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.172371583 (158h22m24.723842044s) [noise=true , address=145.79.227.28:4727 , piece_id=JTVEZRINCPKC7U532YI62SEXI25AXEE6ZBXEXWGTI4IWVEPTVWSQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.173052429 (158h22m24.72452289s) [noise=true , address=145.79.227.20:4439 , piece_id=TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.174665170 (158h22m24.726135621s) [noise=true , address=145.79.227.10:4402 , piece_id=HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.175717564 (158h22m24.727188015s) [noise=true , address=145.79.227.11:4731 , piece_id=2BUZ5Q4KA4MG6EBYHXRICRVAE6MBL6QXMJSIZFV66RBCJ7L6MDKQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.203023491 (158h22m24.754493952s) [noise=true , address=45.91.116.14:4190 , piece_id=X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.211675806 (158h22m24.763146267s) [noise=true , address=145.79.227.21:4473 , piece_id=5WUFKUK7GAWMRPMKCGJD3QAXWLE2AH4SP2WN653JYGWZX5OWWGBA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.241230605 (158h22m24.792701066s) [noise=true , address=145.79.227.18:4005 , piece_id=AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.265904560 (158h22m24.817375011s) [noise=true , address=45.91.116.11:4117 , piece_id=SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.267751306 (158h22m24.819221757s) [noise=true , address=145.79.227.27:4691 , piece_id=AN2YAQFX4APSPXZ37L3THXZDLOU3NZJXZX3GFPQ6GOP2XYMYHJMA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.297086917 (158h22m24.848557379s) [noise=true , address=45.91.116.10:4083 , piece_id=A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.396803830 (158h22m24.948274281s) [noise=true , address=77.90.31.12:4592 , piece_id=DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:51.578031970 (158h22m50.129502431s) [noise=true , address=145.79.227.8:4329 , piece_id=C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:51.578058921 (158h22m50.129529382s) [seg_pos={0 0}, upload=139438]: Segment reached optimal threshold of 13 pieces. Nov 15 22:26:51.578335898 (158h22m50.129806369s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Nov 15 22:26:51.578366757 (158h22m50.129837218s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.578860117 (158h22m50.130330588s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection Nov 15 22:26:51.578892328 (158h22m50.130362789s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.579086789 (158h22m50.13055726s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection Nov 15 22:26:51.579099853 (158h22m50.130570314s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.579150771 (158h22m50.130621232s) [seg_pos={0 0}, upload=139438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 22:26:51.579271630 (158h22m50.130742101s) [upload=139438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 22:26:51.579279916 (158h22m50.130750377s) [upload=139438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 22:26:51.887479665 (158h22m50.438950126s) [upload=139438]: Done uploading segments... Nov 15 22:26:51.887500194 (158h22m50.438970655s) [upload=139438]: Upload finished. err: <nil> Duration: 5.924032563s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.003030791s trace: https://jaeger.dev.storj.io/trace/640930c436a3dd40 FAIL (2025-11-15 22:26:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA) context deadline exceeded; piecestore: (Node ID: 12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, Piece ID: AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA) context deadline exceeded; piecestore: (Node ID: 1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, Piece ID: TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA) context deadline exceeded; piecestore: (Node ID: 12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, Piece ID: DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A) context deadline exceeded ======LOGS====== Nov 15 22:26:23.884265358 (158h22m22.43573582s) [upload=139438]: Got next segment. Inline: false Nov 15 22:26:23.884310675 (158h22m22.435781137s) [upload=139438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 22:26:23.884317198 (158h22m22.435787649s) [upload=139438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 22:26:24.145294674 (158h22m22.696765135s) [seg_pos={0 0}, upload=139438]: Begin upload segment... Nov 15 22:26:24.145353216 (158h22m22.696823667s) [seg_pos={0 0}, upload=139438]: Done begin upload segment. Nov 15 22:26:24.145389284 (158h22m22.696859745s) [noise=true , address=145.79.227.28:4727 , piece_id=JTVEZRINCPKC7U532YI62SEXI25AXEE6ZBXEXWGTI4IWVEPTVWSQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.145586640 (158h22m22.697057091s) [noise=true , address=145.79.227.21:4473 , piece_id=5WUFKUK7GAWMRPMKCGJD3QAXWLE2AH4SP2WN653JYGWZX5OWWGBA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146324976 (158h22m22.697795437s) [noise=true , address=145.79.227.11:4731 , piece_id=2BUZ5Q4KA4MG6EBYHXRICRVAE6MBL6QXMJSIZFV66RBCJ7L6MDKQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146351477 (158h22m22.697821938s) [noise=true , address=45.91.116.11:4117 , piece_id=SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146783870 (158h22m22.698254331s) [noise=true , address=145.79.227.18:4005 , piece_id=AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.146833304 (158h22m22.698303765s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147216514 (158h22m22.698686975s) [noise=true , address=145.79.227.10:4402 , piece_id=HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147263273 (158h22m22.698733734s) [noise=true , address=45.91.116.14:4190 , piece_id=X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147675318 (158h22m22.699145799s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.147754558 (158h22m22.699225019s) [noise=true , address=145.79.227.20:4439 , piece_id=TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148345744 (158h22m22.699816205s) [noise=true , address=45.91.116.10:4083 , piece_id=A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148407592 (158h22m22.699878053s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148798777 (158h22m22.700269228s) [noise=true , address=145.79.227.31:4799 , piece_id=LT3JYVZJ2A3I4VPLCEYONSNIPRBB4E2XSODHWL572IB3NFCM26JA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.148858250 (158h22m22.700328711s) [noise=true , address=145.79.227.27:4691 , piece_id=AN2YAQFX4APSPXZ37L3THXZDLOU3NZJXZX3GFPQ6GOP2XYMYHJMA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.149166006 (158h22m22.700636467s) [noise=true , address=145.79.227.8:4329 , piece_id=C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:24.149373912 (158h22m22.700844373s) [noise=true , address=77.90.31.12:4592 , piece_id=DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA, seg_pos={0 0}, upload=139438]: Uploading piece... Nov 15 22:26:25.877223267 (158h22m24.428693728s) [upload=139438]: Next returned nil segment Nov 15 22:26:25.877245679 (158h22m24.42871614s) [upload=139438]: Waiting for error group managing segments... Nov 15 22:26:26.154633386 (158h22m24.706103847s) [noise=true , address=145.79.227.31:4799 , piece_id=LT3JYVZJ2A3I4VPLCEYONSNIPRBB4E2XSODHWL572IB3NFCM26JA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.172371583 (158h22m24.723842044s) [noise=true , address=145.79.227.28:4727 , piece_id=JTVEZRINCPKC7U532YI62SEXI25AXEE6ZBXEXWGTI4IWVEPTVWSQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.173052429 (158h22m24.72452289s) [noise=true , address=145.79.227.20:4439 , piece_id=TBTUUPTGINRCGEE45OQCJYHPJ4K44WQ2LFVB5PWD4VCELFVMAUWQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.174665170 (158h22m24.726135621s) [noise=true , address=145.79.227.10:4402 , piece_id=HDAPYPR7QYRZGZXUL4CZZJIDUGIUR45SFEXA7FOYTA4ZDASFNU6A, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.175717564 (158h22m24.727188015s) [noise=true , address=145.79.227.11:4731 , piece_id=2BUZ5Q4KA4MG6EBYHXRICRVAE6MBL6QXMJSIZFV66RBCJ7L6MDKQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.203023491 (158h22m24.754493952s) [noise=true , address=45.91.116.14:4190 , piece_id=X6VAUTDFY7JNYILGA65OIWMNP7EV77DJBT4OSR6UUDBYB7E3W5IA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.211675806 (158h22m24.763146267s) [noise=true , address=145.79.227.21:4473 , piece_id=5WUFKUK7GAWMRPMKCGJD3QAXWLE2AH4SP2WN653JYGWZX5OWWGBA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.241230605 (158h22m24.792701066s) [noise=true , address=145.79.227.18:4005 , piece_id=AC4LM2P3F4OTPXWYVYYYDMOA72LSSRWXCFXLEKMXCJVYS52X7AKQ, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.265904560 (158h22m24.817375011s) [noise=true , address=45.91.116.11:4117 , piece_id=SK2MP6UQRZERISSMAS2XFXJ4MMWK7H6QPEF5A3RM47F6FV6NXH2A, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.267751306 (158h22m24.819221757s) [noise=true , address=145.79.227.27:4691 , piece_id=AN2YAQFX4APSPXZ37L3THXZDLOU3NZJXZX3GFPQ6GOP2XYMYHJMA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.297086917 (158h22m24.848557379s) [noise=true , address=45.91.116.10:4083 , piece_id=A37KNXHQ5M3BNCWIAZUVTZC2ANRQJIT3KHJYYILSOYRYSCC5UYRA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:26.396803830 (158h22m24.948274281s) [noise=true , address=77.90.31.12:4592 , piece_id=DJNI77TEU4V5D5UPPKK7KAJHM6VO62OLLF6K6K3RVWQKEDWIIJRA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:51.578031970 (158h22m50.129502431s) [noise=true , address=145.79.227.8:4329 , piece_id=C2R2DAJBEFSSJNNP3YX6RXOXGVRDJWJCHWD2FK4LYIE4O5LE4JPA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: <nil> Nov 15 22:26:51.578058921 (158h22m50.129529382s) [seg_pos={0 0}, upload=139438]: Segment reached optimal threshold of 13 pieces. Nov 15 22:26:51.578335898 (158h22m50.129806369s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Nov 15 22:26:51.578366757 (158h22m50.129837218s) [noise=true , address=145.79.227.14:4837 , piece_id=PGSZCDJKWAHMU6W2MFIBPAFBOHD4PEMMB5YQEJC3KBJCNW6PYMNA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.578860117 (158h22m50.130330588s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52036->145.79.227.16:4910: use of closed network connection Nov 15 22:26:51.578892328 (158h22m50.130362789s) [noise=true , address=145.79.227.16:4910 , piece_id=RVAJVMVRAZF2B4JZOBGRNZ2YN2INGVUISXAWRXDDZQHRRQRYHSQA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.579086789 (158h22m50.13055726s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49934->145.79.227.32:4835: use of closed network connection Nov 15 22:26:51.579099853 (158h22m50.130570314s) [noise=true , address=145.79.227.32:4835 , piece_id=W5HWSJTMI2I4CVTXIKILRJ5FAGWIEQMTE3IX5AGG3FHJ3YHXEZEA, seg_pos={0 0}, upload=139438]: Long tail cancellation due to optimal threshold Nov 15 22:26:51.579150771 (158h22m50.130621232s) [seg_pos={0 0}, upload=139438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 22:26:51.579271630 (158h22m50.130742101s) [upload=139438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 22:26:51.579279916 (158h22m50.130750377s) [upload=139438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 22:26:51.887479665 (158h22m50.438950126s) [upload=139438]: Done uploading segments... Nov 15 22:26:51.887500194 (158h22m50.438970655s) [upload=139438]: Upload finished. err: <nil> Duration: 30.003055217s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 3.547757794s trace: https://jaeger.dev.storj.io/trace/73ad65536e3c10a2 FAIL (2025-11-15 19:24:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.547783411s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 3.551612354s trace: https://jaeger.dev.storj.io/trace/73ad65536e3c10a2 FAIL (2025-11-15 19:23:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.551640388s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 3.468021166s trace: https://jaeger.dev.storj.io/trace/73ad65536e3c10a2 FAIL (2025-11-15 19:23:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.468217157s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 3.479075717s trace: https://jaeger.dev.storj.io/trace/73ad65536e3c10a2 FAIL (2025-11-15 19:22:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.479106605s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 3.443865167s trace: https://jaeger.dev.storj.io/trace/73ad65536e3c10a2 FAIL (2025-11-15 19:21:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.443894532s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 3.582468331s trace: https://jaeger.dev.storj.io/trace/73ad65536e3c10a2 FAIL (2025-11-15 19:20:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.582524488s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 3.370105795s trace: https://jaeger.dev.storj.io/trace/73ad65536e3c10a2 FAIL (2025-11-15 19:19:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.370226704s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 3.495378842s trace: https://jaeger.dev.storj.io/trace/73ad65536e3c10a2 FAIL (2025-11-15 19:18:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.495411996s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 3.415225072s trace: https://jaeger.dev.storj.io/trace/73ad65536e3c10a2 FAIL (2025-11-15 19:17:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.415294133s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 3.582768812s trace: https://jaeger.dev.storj.io/trace/73ad65536e3c10a2 FAIL (2025-11-15 19:16:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.58279377s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 3.455443134s trace: https://jaeger.dev.storj.io/trace/73ad65536e3c10a2 FAIL (2025-11-15 19:15:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.455485284s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 3.514905216s trace: https://jaeger.dev.storj.io/trace/73ad65536e3c10a2 FAIL (2025-11-15 19:15:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.514955773s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 3.502865896s trace: https://jaeger.dev.storj.io/trace/73ad65536e3c10a2 FAIL (2025-11-15 19:14:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.502895662s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 3.531232342s trace: https://jaeger.dev.storj.io/trace/73ad65536e3c10a2 FAIL (2025-11-15 19:14:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.531412495s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 3.520272074s trace: https://jaeger.dev.storj.io/trace/73ad65536e3c10a2 FAIL (2025-11-15 19:13:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.520317069s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 3.500686618s trace: https://jaeger.dev.storj.io/trace/73ad65536e3c10a2 FAIL (2025-11-15 19:13:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.50071837s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 3.45621708s trace: https://jaeger.dev.storj.io/trace/73ad65536e3c10a2 FAIL (2025-11-15 19:12:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.456251054s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 3.548632276s trace: https://jaeger.dev.storj.io/trace/73ad65536e3c10a2 FAIL (2025-11-15 19:11:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.548697973s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 3.578655559s trace: https://jaeger.dev.storj.io/trace/73ad65536e3c10a2 FAIL (2025-11-15 19:10:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.578675518s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 3.471749326s trace: https://jaeger.dev.storj.io/trace/73ad65536e3c10a2 FAIL (2025-11-15 19:10:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.471840259s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 3.502043086s trace: https://jaeger.dev.storj.io/trace/73ad65536e3c10a2 FAIL (2025-11-15 19:09:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.502099905s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 3.602213351s trace: https://jaeger.dev.storj.io/trace/73ad65536e3c10a2 FAIL (2025-11-15 19:08:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.60223941s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 3.594199689s trace: https://jaeger.dev.storj.io/trace/73ad65536e3c10a2 FAIL (2025-11-15 19:07:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.594230389s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 3.482113195s trace: https://jaeger.dev.storj.io/trace/73ad65536e3c10a2 FAIL (2025-11-15 19:06:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.48217363s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 3.596414891s trace: https://jaeger.dev.storj.io/trace/73ad65536e3c10a2 FAIL (2025-11-15 19:06:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.596446601s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 3.534057146s trace: https://jaeger.dev.storj.io/trace/73ad65536e3c10a2 FAIL (2025-11-15 19:05:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.534086352s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 3.43038345s trace: https://jaeger.dev.storj.io/trace/73ad65536e3c10a2 FAIL (2025-11-15 19:04:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.430410732s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 3.484578649s trace: https://jaeger.dev.storj.io/trace/73ad65536e3c10a2 FAIL (2025-11-15 19:04:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.484611382s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 3.586420768s trace: https://jaeger.dev.storj.io/trace/73ad65536e3c10a2 FAIL (2025-11-15 19:03:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.586448662s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 3.555665876s trace: https://jaeger.dev.storj.io/trace/73ad65536e3c10a2 FAIL (2025-11-15 19:02:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.555696294s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)