Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.762051735s trace: https://jaeger.dev.storj.io/trace/4d67fad9e2c6ec76 FAIL (2025-11-14 02:57:40 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.881728976 (114h21m4.433199447s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882337685 (114h21m4.433808146s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882761321 (114h21m4.434231782s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883755024 (114h21m4.435225485s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883875503 (114h21m4.435345974s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.884804482 (114h21m4.436274943s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885204404 (114h21m4.436674865s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885789247 (114h21m4.437259708s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886160905 (114h21m4.437631376s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886974594 (114h21m4.438445055s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887182179 (114h21m4.438652651s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887657485 (114h21m4.439127946s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888130356 (114h21m4.439600817s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888691123 (114h21m4.440161594s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:07.255882067 (114h21m5.807352528s) [upload=100627]: Next returned nil segment Nov 14 02:25:07.255900562 (114h21m5.807371023s) [upload=100627]: Waiting for error group managing segments... Nov 14 02:25:07.629575566 (114h21m6.181046027s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.802313484 (114h21m6.353783945s) [noise=true , address=64.49.13.140:4510 , piece_id=WRZE6WU5UZ56L22NNBV3GVUG3HC52W7AJFP6ULZSLQ25V76JEEQQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.903131683 (114h21m6.454602145s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.907810763 (114h21m6.459281224s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.915612680 (114h21m6.467083141s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.919665296 (114h21m6.471135757s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.927705556 (114h21m6.479176018s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937100136 (114h21m6.488570608s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937995852 (114h21m6.489466323s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.938767291 (114h21m6.490237752s) [noise=true , address=77.90.31.18:4809 , piece_id=FS5BSPEWP734H6KZQ3ILIC7BU532CJ2NHM2K3XL3G5CTM5G26TOA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.975450604 (114h21m6.526921065s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:08.060141452 (114h21m6.611611913s) [noise=true , address=77.90.31.11:4572 , piece_id=LNPRUL5YLDXS3KXUGO72RVPXYHV43Y3PMZZ4LIE3WX2SQW6VHRNA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:09.481028605 (114h21m8.032499066s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364334249 (114h22m2.91580471s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364362523 (114h22m2.915832984s) [seg_pos={0 0}, upload=100627]: Segment reached optimal threshold of 14 pieces. Nov 14 02:26:04.364726596 (114h22m2.916197057s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection Nov 14 02:26:04.364736675 (114h22m2.916207136s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364900176 (114h22m2.916370647s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BZnnbpp63tb9YBYEym3fuvJB4fR1L2GPTpWPbESbVgAXHpa64): protocol: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection Nov 14 02:26:04.364912240 (114h22m2.916382711s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364936345 (114h22m2.916406806s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YdSTc2X61baRotZGev4q25BFkEspbienJtMcpbQvGgbyCRYfx): protocol: read tcp 188.166.225.170:48200->145.79.227.37:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 02:26:04.364944490 (114h22m2.916414941s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365027369 (114h22m2.91649783s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv): protocol: expected piece hash; context canceled; EOF Nov 14 02:26:04.365037989 (114h22m2.91650845s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365060712 (114h22m2.916531173s) [seg_pos={0 0}, upload=100627]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 02:26:04.365180710 (114h22m2.916651172s) [upload=100627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 02:26:04.365188695 (114h22m2.916659156s) [upload=100627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 02:26:04.675897038 (114h22m3.227367509s) [upload=100627]: Done uploading segments... Nov 14 02:26:04.675924961 (114h22m3.227395423s) [upload=100627]: Upload finished. err: <nil> Duration: 5.762083004s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.595323071s trace: https://jaeger.dev.storj.io/trace/4d67fad9e2c6ec76 FAIL (2025-11-14 02:56:35 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.881728976 (114h21m4.433199447s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882337685 (114h21m4.433808146s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882761321 (114h21m4.434231782s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883755024 (114h21m4.435225485s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883875503 (114h21m4.435345974s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.884804482 (114h21m4.436274943s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885204404 (114h21m4.436674865s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885789247 (114h21m4.437259708s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886160905 (114h21m4.437631376s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886974594 (114h21m4.438445055s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887182179 (114h21m4.438652651s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887657485 (114h21m4.439127946s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888130356 (114h21m4.439600817s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888691123 (114h21m4.440161594s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:07.255882067 (114h21m5.807352528s) [upload=100627]: Next returned nil segment Nov 14 02:25:07.255900562 (114h21m5.807371023s) [upload=100627]: Waiting for error group managing segments... Nov 14 02:25:07.629575566 (114h21m6.181046027s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.802313484 (114h21m6.353783945s) [noise=true , address=64.49.13.140:4510 , piece_id=WRZE6WU5UZ56L22NNBV3GVUG3HC52W7AJFP6ULZSLQ25V76JEEQQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.903131683 (114h21m6.454602145s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.907810763 (114h21m6.459281224s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.915612680 (114h21m6.467083141s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.919665296 (114h21m6.471135757s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.927705556 (114h21m6.479176018s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937100136 (114h21m6.488570608s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937995852 (114h21m6.489466323s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.938767291 (114h21m6.490237752s) [noise=true , address=77.90.31.18:4809 , piece_id=FS5BSPEWP734H6KZQ3ILIC7BU532CJ2NHM2K3XL3G5CTM5G26TOA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.975450604 (114h21m6.526921065s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:08.060141452 (114h21m6.611611913s) [noise=true , address=77.90.31.11:4572 , piece_id=LNPRUL5YLDXS3KXUGO72RVPXYHV43Y3PMZZ4LIE3WX2SQW6VHRNA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:09.481028605 (114h21m8.032499066s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364334249 (114h22m2.91580471s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364362523 (114h22m2.915832984s) [seg_pos={0 0}, upload=100627]: Segment reached optimal threshold of 14 pieces. Nov 14 02:26:04.364726596 (114h22m2.916197057s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection Nov 14 02:26:04.364736675 (114h22m2.916207136s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364900176 (114h22m2.916370647s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BZnnbpp63tb9YBYEym3fuvJB4fR1L2GPTpWPbESbVgAXHpa64): protocol: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection Nov 14 02:26:04.364912240 (114h22m2.916382711s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364936345 (114h22m2.916406806s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YdSTc2X61baRotZGev4q25BFkEspbienJtMcpbQvGgbyCRYfx): protocol: read tcp 188.166.225.170:48200->145.79.227.37:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 02:26:04.364944490 (114h22m2.916414941s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365027369 (114h22m2.91649783s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv): protocol: expected piece hash; context canceled; EOF Nov 14 02:26:04.365037989 (114h22m2.91650845s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365060712 (114h22m2.916531173s) [seg_pos={0 0}, upload=100627]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 02:26:04.365180710 (114h22m2.916651172s) [upload=100627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 02:26:04.365188695 (114h22m2.916659156s) [upload=100627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 02:26:04.675897038 (114h22m3.227367509s) [upload=100627]: Done uploading segments... Nov 14 02:26:04.675924961 (114h22m3.227395423s) [upload=100627]: Upload finished. err: <nil> Duration: 5.595430455s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.584653846s trace: https://jaeger.dev.storj.io/trace/4d67fad9e2c6ec76 FAIL (2025-11-14 02:55:49 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.881728976 (114h21m4.433199447s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882337685 (114h21m4.433808146s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882761321 (114h21m4.434231782s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883755024 (114h21m4.435225485s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883875503 (114h21m4.435345974s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.884804482 (114h21m4.436274943s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885204404 (114h21m4.436674865s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885789247 (114h21m4.437259708s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886160905 (114h21m4.437631376s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886974594 (114h21m4.438445055s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887182179 (114h21m4.438652651s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887657485 (114h21m4.439127946s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888130356 (114h21m4.439600817s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888691123 (114h21m4.440161594s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:07.255882067 (114h21m5.807352528s) [upload=100627]: Next returned nil segment Nov 14 02:25:07.255900562 (114h21m5.807371023s) [upload=100627]: Waiting for error group managing segments... Nov 14 02:25:07.629575566 (114h21m6.181046027s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.802313484 (114h21m6.353783945s) [noise=true , address=64.49.13.140:4510 , piece_id=WRZE6WU5UZ56L22NNBV3GVUG3HC52W7AJFP6ULZSLQ25V76JEEQQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.903131683 (114h21m6.454602145s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.907810763 (114h21m6.459281224s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.915612680 (114h21m6.467083141s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.919665296 (114h21m6.471135757s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.927705556 (114h21m6.479176018s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937100136 (114h21m6.488570608s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937995852 (114h21m6.489466323s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.938767291 (114h21m6.490237752s) [noise=true , address=77.90.31.18:4809 , piece_id=FS5BSPEWP734H6KZQ3ILIC7BU532CJ2NHM2K3XL3G5CTM5G26TOA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.975450604 (114h21m6.526921065s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:08.060141452 (114h21m6.611611913s) [noise=true , address=77.90.31.11:4572 , piece_id=LNPRUL5YLDXS3KXUGO72RVPXYHV43Y3PMZZ4LIE3WX2SQW6VHRNA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:09.481028605 (114h21m8.032499066s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364334249 (114h22m2.91580471s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364362523 (114h22m2.915832984s) [seg_pos={0 0}, upload=100627]: Segment reached optimal threshold of 14 pieces. Nov 14 02:26:04.364726596 (114h22m2.916197057s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection Nov 14 02:26:04.364736675 (114h22m2.916207136s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364900176 (114h22m2.916370647s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BZnnbpp63tb9YBYEym3fuvJB4fR1L2GPTpWPbESbVgAXHpa64): protocol: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection Nov 14 02:26:04.364912240 (114h22m2.916382711s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364936345 (114h22m2.916406806s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YdSTc2X61baRotZGev4q25BFkEspbienJtMcpbQvGgbyCRYfx): protocol: read tcp 188.166.225.170:48200->145.79.227.37:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 02:26:04.364944490 (114h22m2.916414941s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365027369 (114h22m2.91649783s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv): protocol: expected piece hash; context canceled; EOF Nov 14 02:26:04.365037989 (114h22m2.91650845s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365060712 (114h22m2.916531173s) [seg_pos={0 0}, upload=100627]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 02:26:04.365180710 (114h22m2.916651172s) [upload=100627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 02:26:04.365188695 (114h22m2.916659156s) [upload=100627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 02:26:04.675897038 (114h22m3.227367509s) [upload=100627]: Done uploading segments... Nov 14 02:26:04.675924961 (114h22m3.227395423s) [upload=100627]: Upload finished. err: <nil> Duration: 5.584700404s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.589355146s trace: https://jaeger.dev.storj.io/trace/4d67fad9e2c6ec76 FAIL (2025-11-14 02:54:56 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.881728976 (114h21m4.433199447s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882337685 (114h21m4.433808146s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882761321 (114h21m4.434231782s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883755024 (114h21m4.435225485s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883875503 (114h21m4.435345974s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.884804482 (114h21m4.436274943s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885204404 (114h21m4.436674865s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885789247 (114h21m4.437259708s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886160905 (114h21m4.437631376s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886974594 (114h21m4.438445055s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887182179 (114h21m4.438652651s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887657485 (114h21m4.439127946s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888130356 (114h21m4.439600817s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888691123 (114h21m4.440161594s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:07.255882067 (114h21m5.807352528s) [upload=100627]: Next returned nil segment Nov 14 02:25:07.255900562 (114h21m5.807371023s) [upload=100627]: Waiting for error group managing segments... Nov 14 02:25:07.629575566 (114h21m6.181046027s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.802313484 (114h21m6.353783945s) [noise=true , address=64.49.13.140:4510 , piece_id=WRZE6WU5UZ56L22NNBV3GVUG3HC52W7AJFP6ULZSLQ25V76JEEQQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.903131683 (114h21m6.454602145s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.907810763 (114h21m6.459281224s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.915612680 (114h21m6.467083141s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.919665296 (114h21m6.471135757s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.927705556 (114h21m6.479176018s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937100136 (114h21m6.488570608s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937995852 (114h21m6.489466323s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.938767291 (114h21m6.490237752s) [noise=true , address=77.90.31.18:4809 , piece_id=FS5BSPEWP734H6KZQ3ILIC7BU532CJ2NHM2K3XL3G5CTM5G26TOA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.975450604 (114h21m6.526921065s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:08.060141452 (114h21m6.611611913s) [noise=true , address=77.90.31.11:4572 , piece_id=LNPRUL5YLDXS3KXUGO72RVPXYHV43Y3PMZZ4LIE3WX2SQW6VHRNA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:09.481028605 (114h21m8.032499066s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364334249 (114h22m2.91580471s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364362523 (114h22m2.915832984s) [seg_pos={0 0}, upload=100627]: Segment reached optimal threshold of 14 pieces. Nov 14 02:26:04.364726596 (114h22m2.916197057s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection Nov 14 02:26:04.364736675 (114h22m2.916207136s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364900176 (114h22m2.916370647s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BZnnbpp63tb9YBYEym3fuvJB4fR1L2GPTpWPbESbVgAXHpa64): protocol: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection Nov 14 02:26:04.364912240 (114h22m2.916382711s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364936345 (114h22m2.916406806s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YdSTc2X61baRotZGev4q25BFkEspbienJtMcpbQvGgbyCRYfx): protocol: read tcp 188.166.225.170:48200->145.79.227.37:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 02:26:04.364944490 (114h22m2.916414941s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365027369 (114h22m2.91649783s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv): protocol: expected piece hash; context canceled; EOF Nov 14 02:26:04.365037989 (114h22m2.91650845s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365060712 (114h22m2.916531173s) [seg_pos={0 0}, upload=100627]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 02:26:04.365180710 (114h22m2.916651172s) [upload=100627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 02:26:04.365188695 (114h22m2.916659156s) [upload=100627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 02:26:04.675897038 (114h22m3.227367509s) [upload=100627]: Done uploading segments... Nov 14 02:26:04.675924961 (114h22m3.227395423s) [upload=100627]: Upload finished. err: <nil> Duration: 5.589381887s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.853223162s trace: https://jaeger.dev.storj.io/trace/4d67fad9e2c6ec76 FAIL (2025-11-14 02:54:32 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.881728976 (114h21m4.433199447s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882337685 (114h21m4.433808146s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882761321 (114h21m4.434231782s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883755024 (114h21m4.435225485s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883875503 (114h21m4.435345974s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.884804482 (114h21m4.436274943s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885204404 (114h21m4.436674865s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885789247 (114h21m4.437259708s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886160905 (114h21m4.437631376s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886974594 (114h21m4.438445055s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887182179 (114h21m4.438652651s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887657485 (114h21m4.439127946s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888130356 (114h21m4.439600817s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888691123 (114h21m4.440161594s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:07.255882067 (114h21m5.807352528s) [upload=100627]: Next returned nil segment Nov 14 02:25:07.255900562 (114h21m5.807371023s) [upload=100627]: Waiting for error group managing segments... Nov 14 02:25:07.629575566 (114h21m6.181046027s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.802313484 (114h21m6.353783945s) [noise=true , address=64.49.13.140:4510 , piece_id=WRZE6WU5UZ56L22NNBV3GVUG3HC52W7AJFP6ULZSLQ25V76JEEQQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.903131683 (114h21m6.454602145s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.907810763 (114h21m6.459281224s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.915612680 (114h21m6.467083141s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.919665296 (114h21m6.471135757s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.927705556 (114h21m6.479176018s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937100136 (114h21m6.488570608s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937995852 (114h21m6.489466323s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.938767291 (114h21m6.490237752s) [noise=true , address=77.90.31.18:4809 , piece_id=FS5BSPEWP734H6KZQ3ILIC7BU532CJ2NHM2K3XL3G5CTM5G26TOA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.975450604 (114h21m6.526921065s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:08.060141452 (114h21m6.611611913s) [noise=true , address=77.90.31.11:4572 , piece_id=LNPRUL5YLDXS3KXUGO72RVPXYHV43Y3PMZZ4LIE3WX2SQW6VHRNA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:09.481028605 (114h21m8.032499066s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364334249 (114h22m2.91580471s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364362523 (114h22m2.915832984s) [seg_pos={0 0}, upload=100627]: Segment reached optimal threshold of 14 pieces. Nov 14 02:26:04.364726596 (114h22m2.916197057s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection Nov 14 02:26:04.364736675 (114h22m2.916207136s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364900176 (114h22m2.916370647s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BZnnbpp63tb9YBYEym3fuvJB4fR1L2GPTpWPbESbVgAXHpa64): protocol: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection Nov 14 02:26:04.364912240 (114h22m2.916382711s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364936345 (114h22m2.916406806s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YdSTc2X61baRotZGev4q25BFkEspbienJtMcpbQvGgbyCRYfx): protocol: read tcp 188.166.225.170:48200->145.79.227.37:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 02:26:04.364944490 (114h22m2.916414941s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365027369 (114h22m2.91649783s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv): protocol: expected piece hash; context canceled; EOF Nov 14 02:26:04.365037989 (114h22m2.91650845s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365060712 (114h22m2.916531173s) [seg_pos={0 0}, upload=100627]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 02:26:04.365180710 (114h22m2.916651172s) [upload=100627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 02:26:04.365188695 (114h22m2.916659156s) [upload=100627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 02:26:04.675897038 (114h22m3.227367509s) [upload=100627]: Done uploading segments... Nov 14 02:26:04.675924961 (114h22m3.227395423s) [upload=100627]: Upload finished. err: <nil> Duration: 5.853252721s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.59309418s trace: https://jaeger.dev.storj.io/trace/4d67fad9e2c6ec76 FAIL (2025-11-14 02:54:06 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.881728976 (114h21m4.433199447s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882337685 (114h21m4.433808146s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882761321 (114h21m4.434231782s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883755024 (114h21m4.435225485s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883875503 (114h21m4.435345974s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.884804482 (114h21m4.436274943s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885204404 (114h21m4.436674865s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885789247 (114h21m4.437259708s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886160905 (114h21m4.437631376s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886974594 (114h21m4.438445055s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887182179 (114h21m4.438652651s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887657485 (114h21m4.439127946s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888130356 (114h21m4.439600817s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888691123 (114h21m4.440161594s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:07.255882067 (114h21m5.807352528s) [upload=100627]: Next returned nil segment Nov 14 02:25:07.255900562 (114h21m5.807371023s) [upload=100627]: Waiting for error group managing segments... Nov 14 02:25:07.629575566 (114h21m6.181046027s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.802313484 (114h21m6.353783945s) [noise=true , address=64.49.13.140:4510 , piece_id=WRZE6WU5UZ56L22NNBV3GVUG3HC52W7AJFP6ULZSLQ25V76JEEQQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.903131683 (114h21m6.454602145s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.907810763 (114h21m6.459281224s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.915612680 (114h21m6.467083141s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.919665296 (114h21m6.471135757s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.927705556 (114h21m6.479176018s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937100136 (114h21m6.488570608s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937995852 (114h21m6.489466323s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.938767291 (114h21m6.490237752s) [noise=true , address=77.90.31.18:4809 , piece_id=FS5BSPEWP734H6KZQ3ILIC7BU532CJ2NHM2K3XL3G5CTM5G26TOA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.975450604 (114h21m6.526921065s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:08.060141452 (114h21m6.611611913s) [noise=true , address=77.90.31.11:4572 , piece_id=LNPRUL5YLDXS3KXUGO72RVPXYHV43Y3PMZZ4LIE3WX2SQW6VHRNA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:09.481028605 (114h21m8.032499066s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364334249 (114h22m2.91580471s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364362523 (114h22m2.915832984s) [seg_pos={0 0}, upload=100627]: Segment reached optimal threshold of 14 pieces. Nov 14 02:26:04.364726596 (114h22m2.916197057s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection Nov 14 02:26:04.364736675 (114h22m2.916207136s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364900176 (114h22m2.916370647s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BZnnbpp63tb9YBYEym3fuvJB4fR1L2GPTpWPbESbVgAXHpa64): protocol: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection Nov 14 02:26:04.364912240 (114h22m2.916382711s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364936345 (114h22m2.916406806s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YdSTc2X61baRotZGev4q25BFkEspbienJtMcpbQvGgbyCRYfx): protocol: read tcp 188.166.225.170:48200->145.79.227.37:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 02:26:04.364944490 (114h22m2.916414941s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365027369 (114h22m2.91649783s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv): protocol: expected piece hash; context canceled; EOF Nov 14 02:26:04.365037989 (114h22m2.91650845s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365060712 (114h22m2.916531173s) [seg_pos={0 0}, upload=100627]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 02:26:04.365180710 (114h22m2.916651172s) [upload=100627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 02:26:04.365188695 (114h22m2.916659156s) [upload=100627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 02:26:04.675897038 (114h22m3.227367509s) [upload=100627]: Done uploading segments... Nov 14 02:26:04.675924961 (114h22m3.227395423s) [upload=100627]: Upload finished. err: <nil> Duration: 5.59313076s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.880447704s trace: https://jaeger.dev.storj.io/trace/4d67fad9e2c6ec76 FAIL (2025-11-14 02:53:09 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.881728976 (114h21m4.433199447s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882337685 (114h21m4.433808146s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882761321 (114h21m4.434231782s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883755024 (114h21m4.435225485s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883875503 (114h21m4.435345974s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.884804482 (114h21m4.436274943s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885204404 (114h21m4.436674865s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885789247 (114h21m4.437259708s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886160905 (114h21m4.437631376s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886974594 (114h21m4.438445055s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887182179 (114h21m4.438652651s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887657485 (114h21m4.439127946s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888130356 (114h21m4.439600817s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888691123 (114h21m4.440161594s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:07.255882067 (114h21m5.807352528s) [upload=100627]: Next returned nil segment Nov 14 02:25:07.255900562 (114h21m5.807371023s) [upload=100627]: Waiting for error group managing segments... Nov 14 02:25:07.629575566 (114h21m6.181046027s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.802313484 (114h21m6.353783945s) [noise=true , address=64.49.13.140:4510 , piece_id=WRZE6WU5UZ56L22NNBV3GVUG3HC52W7AJFP6ULZSLQ25V76JEEQQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.903131683 (114h21m6.454602145s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.907810763 (114h21m6.459281224s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.915612680 (114h21m6.467083141s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.919665296 (114h21m6.471135757s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.927705556 (114h21m6.479176018s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937100136 (114h21m6.488570608s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937995852 (114h21m6.489466323s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.938767291 (114h21m6.490237752s) [noise=true , address=77.90.31.18:4809 , piece_id=FS5BSPEWP734H6KZQ3ILIC7BU532CJ2NHM2K3XL3G5CTM5G26TOA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.975450604 (114h21m6.526921065s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:08.060141452 (114h21m6.611611913s) [noise=true , address=77.90.31.11:4572 , piece_id=LNPRUL5YLDXS3KXUGO72RVPXYHV43Y3PMZZ4LIE3WX2SQW6VHRNA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:09.481028605 (114h21m8.032499066s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364334249 (114h22m2.91580471s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364362523 (114h22m2.915832984s) [seg_pos={0 0}, upload=100627]: Segment reached optimal threshold of 14 pieces. Nov 14 02:26:04.364726596 (114h22m2.916197057s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection Nov 14 02:26:04.364736675 (114h22m2.916207136s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364900176 (114h22m2.916370647s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BZnnbpp63tb9YBYEym3fuvJB4fR1L2GPTpWPbESbVgAXHpa64): protocol: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection Nov 14 02:26:04.364912240 (114h22m2.916382711s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364936345 (114h22m2.916406806s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YdSTc2X61baRotZGev4q25BFkEspbienJtMcpbQvGgbyCRYfx): protocol: read tcp 188.166.225.170:48200->145.79.227.37:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 02:26:04.364944490 (114h22m2.916414941s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365027369 (114h22m2.91649783s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv): protocol: expected piece hash; context canceled; EOF Nov 14 02:26:04.365037989 (114h22m2.91650845s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365060712 (114h22m2.916531173s) [seg_pos={0 0}, upload=100627]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 02:26:04.365180710 (114h22m2.916651172s) [upload=100627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 02:26:04.365188695 (114h22m2.916659156s) [upload=100627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 02:26:04.675897038 (114h22m3.227367509s) [upload=100627]: Done uploading segments... Nov 14 02:26:04.675924961 (114h22m3.227395423s) [upload=100627]: Upload finished. err: <nil> Duration: 5.880488903s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.857602063s trace: https://jaeger.dev.storj.io/trace/4d67fad9e2c6ec76 FAIL (2025-11-14 02:52:52 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.881728976 (114h21m4.433199447s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882337685 (114h21m4.433808146s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882761321 (114h21m4.434231782s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883755024 (114h21m4.435225485s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883875503 (114h21m4.435345974s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.884804482 (114h21m4.436274943s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885204404 (114h21m4.436674865s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885789247 (114h21m4.437259708s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886160905 (114h21m4.437631376s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886974594 (114h21m4.438445055s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887182179 (114h21m4.438652651s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887657485 (114h21m4.439127946s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888130356 (114h21m4.439600817s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888691123 (114h21m4.440161594s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:07.255882067 (114h21m5.807352528s) [upload=100627]: Next returned nil segment Nov 14 02:25:07.255900562 (114h21m5.807371023s) [upload=100627]: Waiting for error group managing segments... Nov 14 02:25:07.629575566 (114h21m6.181046027s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.802313484 (114h21m6.353783945s) [noise=true , address=64.49.13.140:4510 , piece_id=WRZE6WU5UZ56L22NNBV3GVUG3HC52W7AJFP6ULZSLQ25V76JEEQQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.903131683 (114h21m6.454602145s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.907810763 (114h21m6.459281224s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.915612680 (114h21m6.467083141s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.919665296 (114h21m6.471135757s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.927705556 (114h21m6.479176018s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937100136 (114h21m6.488570608s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937995852 (114h21m6.489466323s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.938767291 (114h21m6.490237752s) [noise=true , address=77.90.31.18:4809 , piece_id=FS5BSPEWP734H6KZQ3ILIC7BU532CJ2NHM2K3XL3G5CTM5G26TOA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.975450604 (114h21m6.526921065s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:08.060141452 (114h21m6.611611913s) [noise=true , address=77.90.31.11:4572 , piece_id=LNPRUL5YLDXS3KXUGO72RVPXYHV43Y3PMZZ4LIE3WX2SQW6VHRNA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:09.481028605 (114h21m8.032499066s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364334249 (114h22m2.91580471s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364362523 (114h22m2.915832984s) [seg_pos={0 0}, upload=100627]: Segment reached optimal threshold of 14 pieces. Nov 14 02:26:04.364726596 (114h22m2.916197057s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection Nov 14 02:26:04.364736675 (114h22m2.916207136s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364900176 (114h22m2.916370647s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BZnnbpp63tb9YBYEym3fuvJB4fR1L2GPTpWPbESbVgAXHpa64): protocol: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection Nov 14 02:26:04.364912240 (114h22m2.916382711s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364936345 (114h22m2.916406806s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YdSTc2X61baRotZGev4q25BFkEspbienJtMcpbQvGgbyCRYfx): protocol: read tcp 188.166.225.170:48200->145.79.227.37:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 02:26:04.364944490 (114h22m2.916414941s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365027369 (114h22m2.91649783s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv): protocol: expected piece hash; context canceled; EOF Nov 14 02:26:04.365037989 (114h22m2.91650845s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365060712 (114h22m2.916531173s) [seg_pos={0 0}, upload=100627]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 02:26:04.365180710 (114h22m2.916651172s) [upload=100627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 02:26:04.365188695 (114h22m2.916659156s) [upload=100627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 02:26:04.675897038 (114h22m3.227367509s) [upload=100627]: Done uploading segments... Nov 14 02:26:04.675924961 (114h22m3.227395423s) [upload=100627]: Upload finished. err: <nil> Duration: 5.857630969s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [X..............] duration: 65.22918ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/496c88a2aec2b412 FAIL (2025-11-14 02:52:28 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:54352->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 65.257594ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.791815423s trace: https://jaeger.dev.storj.io/trace/4d67fad9e2c6ec76 FAIL (2025-11-14 02:52:02 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.881728976 (114h21m4.433199447s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882337685 (114h21m4.433808146s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882761321 (114h21m4.434231782s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883755024 (114h21m4.435225485s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883875503 (114h21m4.435345974s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.884804482 (114h21m4.436274943s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885204404 (114h21m4.436674865s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885789247 (114h21m4.437259708s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886160905 (114h21m4.437631376s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886974594 (114h21m4.438445055s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887182179 (114h21m4.438652651s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887657485 (114h21m4.439127946s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888130356 (114h21m4.439600817s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888691123 (114h21m4.440161594s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:07.255882067 (114h21m5.807352528s) [upload=100627]: Next returned nil segment Nov 14 02:25:07.255900562 (114h21m5.807371023s) [upload=100627]: Waiting for error group managing segments... Nov 14 02:25:07.629575566 (114h21m6.181046027s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.802313484 (114h21m6.353783945s) [noise=true , address=64.49.13.140:4510 , piece_id=WRZE6WU5UZ56L22NNBV3GVUG3HC52W7AJFP6ULZSLQ25V76JEEQQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.903131683 (114h21m6.454602145s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.907810763 (114h21m6.459281224s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.915612680 (114h21m6.467083141s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.919665296 (114h21m6.471135757s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.927705556 (114h21m6.479176018s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937100136 (114h21m6.488570608s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937995852 (114h21m6.489466323s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.938767291 (114h21m6.490237752s) [noise=true , address=77.90.31.18:4809 , piece_id=FS5BSPEWP734H6KZQ3ILIC7BU532CJ2NHM2K3XL3G5CTM5G26TOA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.975450604 (114h21m6.526921065s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:08.060141452 (114h21m6.611611913s) [noise=true , address=77.90.31.11:4572 , piece_id=LNPRUL5YLDXS3KXUGO72RVPXYHV43Y3PMZZ4LIE3WX2SQW6VHRNA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:09.481028605 (114h21m8.032499066s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364334249 (114h22m2.91580471s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364362523 (114h22m2.915832984s) [seg_pos={0 0}, upload=100627]: Segment reached optimal threshold of 14 pieces. Nov 14 02:26:04.364726596 (114h22m2.916197057s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection Nov 14 02:26:04.364736675 (114h22m2.916207136s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364900176 (114h22m2.916370647s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BZnnbpp63tb9YBYEym3fuvJB4fR1L2GPTpWPbESbVgAXHpa64): protocol: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection Nov 14 02:26:04.364912240 (114h22m2.916382711s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364936345 (114h22m2.916406806s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YdSTc2X61baRotZGev4q25BFkEspbienJtMcpbQvGgbyCRYfx): protocol: read tcp 188.166.225.170:48200->145.79.227.37:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 02:26:04.364944490 (114h22m2.916414941s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365027369 (114h22m2.91649783s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv): protocol: expected piece hash; context canceled; EOF Nov 14 02:26:04.365037989 (114h22m2.91650845s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365060712 (114h22m2.916531173s) [seg_pos={0 0}, upload=100627]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 02:26:04.365180710 (114h22m2.916651172s) [upload=100627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 02:26:04.365188695 (114h22m2.916659156s) [upload=100627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 02:26:04.675897038 (114h22m3.227367509s) [upload=100627]: Done uploading segments... Nov 14 02:26:04.675924961 (114h22m3.227395423s) [upload=100627]: Upload finished. err: <nil> Duration: 5.791847203s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.152502103s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:51:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.15260101s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.X.............] duration: 51.020704ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/496c88a2aec2b412 FAIL (2025-11-14 02:51:26 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:54352->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 51.047955ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.590428573s trace: https://jaeger.dev.storj.io/trace/4d67fad9e2c6ec76 FAIL (2025-11-14 02:51:22 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.881728976 (114h21m4.433199447s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882337685 (114h21m4.433808146s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882761321 (114h21m4.434231782s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883755024 (114h21m4.435225485s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883875503 (114h21m4.435345974s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.884804482 (114h21m4.436274943s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885204404 (114h21m4.436674865s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885789247 (114h21m4.437259708s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886160905 (114h21m4.437631376s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886974594 (114h21m4.438445055s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887182179 (114h21m4.438652651s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887657485 (114h21m4.439127946s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888130356 (114h21m4.439600817s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888691123 (114h21m4.440161594s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:07.255882067 (114h21m5.807352528s) [upload=100627]: Next returned nil segment Nov 14 02:25:07.255900562 (114h21m5.807371023s) [upload=100627]: Waiting for error group managing segments... Nov 14 02:25:07.629575566 (114h21m6.181046027s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.802313484 (114h21m6.353783945s) [noise=true , address=64.49.13.140:4510 , piece_id=WRZE6WU5UZ56L22NNBV3GVUG3HC52W7AJFP6ULZSLQ25V76JEEQQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.903131683 (114h21m6.454602145s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.907810763 (114h21m6.459281224s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.915612680 (114h21m6.467083141s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.919665296 (114h21m6.471135757s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.927705556 (114h21m6.479176018s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937100136 (114h21m6.488570608s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937995852 (114h21m6.489466323s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.938767291 (114h21m6.490237752s) [noise=true , address=77.90.31.18:4809 , piece_id=FS5BSPEWP734H6KZQ3ILIC7BU532CJ2NHM2K3XL3G5CTM5G26TOA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.975450604 (114h21m6.526921065s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:08.060141452 (114h21m6.611611913s) [noise=true , address=77.90.31.11:4572 , piece_id=LNPRUL5YLDXS3KXUGO72RVPXYHV43Y3PMZZ4LIE3WX2SQW6VHRNA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:09.481028605 (114h21m8.032499066s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364334249 (114h22m2.91580471s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364362523 (114h22m2.915832984s) [seg_pos={0 0}, upload=100627]: Segment reached optimal threshold of 14 pieces. Nov 14 02:26:04.364726596 (114h22m2.916197057s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection Nov 14 02:26:04.364736675 (114h22m2.916207136s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364900176 (114h22m2.916370647s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BZnnbpp63tb9YBYEym3fuvJB4fR1L2GPTpWPbESbVgAXHpa64): protocol: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection Nov 14 02:26:04.364912240 (114h22m2.916382711s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364936345 (114h22m2.916406806s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YdSTc2X61baRotZGev4q25BFkEspbienJtMcpbQvGgbyCRYfx): protocol: read tcp 188.166.225.170:48200->145.79.227.37:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 02:26:04.364944490 (114h22m2.916414941s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365027369 (114h22m2.91649783s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv): protocol: expected piece hash; context canceled; EOF Nov 14 02:26:04.365037989 (114h22m2.91650845s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365060712 (114h22m2.916531173s) [seg_pos={0 0}, upload=100627]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 02:26:04.365180710 (114h22m2.916651172s) [upload=100627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 02:26:04.365188695 (114h22m2.916659156s) [upload=100627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 02:26:04.675897038 (114h22m3.227367509s) [upload=100627]: Done uploading segments... Nov 14 02:26:04.675924961 (114h22m3.227395423s) [upload=100627]: Upload finished. err: <nil> Duration: 5.59046399s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 6.109547306s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:50:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.109582062s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.800858564s trace: https://jaeger.dev.storj.io/trace/4d67fad9e2c6ec76 FAIL (2025-11-14 02:50:54 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.881728976 (114h21m4.433199447s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882337685 (114h21m4.433808146s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882761321 (114h21m4.434231782s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883755024 (114h21m4.435225485s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883875503 (114h21m4.435345974s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.884804482 (114h21m4.436274943s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885204404 (114h21m4.436674865s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885789247 (114h21m4.437259708s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886160905 (114h21m4.437631376s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886974594 (114h21m4.438445055s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887182179 (114h21m4.438652651s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887657485 (114h21m4.439127946s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888130356 (114h21m4.439600817s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888691123 (114h21m4.440161594s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:07.255882067 (114h21m5.807352528s) [upload=100627]: Next returned nil segment Nov 14 02:25:07.255900562 (114h21m5.807371023s) [upload=100627]: Waiting for error group managing segments... Nov 14 02:25:07.629575566 (114h21m6.181046027s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.802313484 (114h21m6.353783945s) [noise=true , address=64.49.13.140:4510 , piece_id=WRZE6WU5UZ56L22NNBV3GVUG3HC52W7AJFP6ULZSLQ25V76JEEQQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.903131683 (114h21m6.454602145s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.907810763 (114h21m6.459281224s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.915612680 (114h21m6.467083141s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.919665296 (114h21m6.471135757s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.927705556 (114h21m6.479176018s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937100136 (114h21m6.488570608s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937995852 (114h21m6.489466323s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.938767291 (114h21m6.490237752s) [noise=true , address=77.90.31.18:4809 , piece_id=FS5BSPEWP734H6KZQ3ILIC7BU532CJ2NHM2K3XL3G5CTM5G26TOA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.975450604 (114h21m6.526921065s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:08.060141452 (114h21m6.611611913s) [noise=true , address=77.90.31.11:4572 , piece_id=LNPRUL5YLDXS3KXUGO72RVPXYHV43Y3PMZZ4LIE3WX2SQW6VHRNA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:09.481028605 (114h21m8.032499066s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364334249 (114h22m2.91580471s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364362523 (114h22m2.915832984s) [seg_pos={0 0}, upload=100627]: Segment reached optimal threshold of 14 pieces. Nov 14 02:26:04.364726596 (114h22m2.916197057s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection Nov 14 02:26:04.364736675 (114h22m2.916207136s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364900176 (114h22m2.916370647s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BZnnbpp63tb9YBYEym3fuvJB4fR1L2GPTpWPbESbVgAXHpa64): protocol: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection Nov 14 02:26:04.364912240 (114h22m2.916382711s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364936345 (114h22m2.916406806s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YdSTc2X61baRotZGev4q25BFkEspbienJtMcpbQvGgbyCRYfx): protocol: read tcp 188.166.225.170:48200->145.79.227.37:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 02:26:04.364944490 (114h22m2.916414941s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365027369 (114h22m2.91649783s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv): protocol: expected piece hash; context canceled; EOF Nov 14 02:26:04.365037989 (114h22m2.91650845s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365060712 (114h22m2.916531173s) [seg_pos={0 0}, upload=100627]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 02:26:04.365180710 (114h22m2.916651172s) [upload=100627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 02:26:04.365188695 (114h22m2.916659156s) [upload=100627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 02:26:04.675897038 (114h22m3.227367509s) [upload=100627]: Done uploading segments... Nov 14 02:26:04.675924961 (114h22m3.227395423s) [upload=100627]: Upload finished. err: <nil> Duration: 5.80088893s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.XX.........................................................] duration: 6.04837159s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:50:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.048399792s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..X............] duration: 79.234929ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/496c88a2aec2b412 FAIL (2025-11-14 02:50:13 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:54352->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 79.266699ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.885292904s trace: https://jaeger.dev.storj.io/trace/4d67fad9e2c6ec76 FAIL (2025-11-14 02:50:11 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.881728976 (114h21m4.433199447s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882337685 (114h21m4.433808146s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882761321 (114h21m4.434231782s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883755024 (114h21m4.435225485s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883875503 (114h21m4.435345974s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.884804482 (114h21m4.436274943s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885204404 (114h21m4.436674865s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885789247 (114h21m4.437259708s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886160905 (114h21m4.437631376s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886974594 (114h21m4.438445055s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887182179 (114h21m4.438652651s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887657485 (114h21m4.439127946s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888130356 (114h21m4.439600817s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888691123 (114h21m4.440161594s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:07.255882067 (114h21m5.807352528s) [upload=100627]: Next returned nil segment Nov 14 02:25:07.255900562 (114h21m5.807371023s) [upload=100627]: Waiting for error group managing segments... Nov 14 02:25:07.629575566 (114h21m6.181046027s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.802313484 (114h21m6.353783945s) [noise=true , address=64.49.13.140:4510 , piece_id=WRZE6WU5UZ56L22NNBV3GVUG3HC52W7AJFP6ULZSLQ25V76JEEQQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.903131683 (114h21m6.454602145s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.907810763 (114h21m6.459281224s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.915612680 (114h21m6.467083141s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.919665296 (114h21m6.471135757s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.927705556 (114h21m6.479176018s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937100136 (114h21m6.488570608s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937995852 (114h21m6.489466323s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.938767291 (114h21m6.490237752s) [noise=true , address=77.90.31.18:4809 , piece_id=FS5BSPEWP734H6KZQ3ILIC7BU532CJ2NHM2K3XL3G5CTM5G26TOA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.975450604 (114h21m6.526921065s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:08.060141452 (114h21m6.611611913s) [noise=true , address=77.90.31.11:4572 , piece_id=LNPRUL5YLDXS3KXUGO72RVPXYHV43Y3PMZZ4LIE3WX2SQW6VHRNA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:09.481028605 (114h21m8.032499066s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364334249 (114h22m2.91580471s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364362523 (114h22m2.915832984s) [seg_pos={0 0}, upload=100627]: Segment reached optimal threshold of 14 pieces. Nov 14 02:26:04.364726596 (114h22m2.916197057s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection Nov 14 02:26:04.364736675 (114h22m2.916207136s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364900176 (114h22m2.916370647s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BZnnbpp63tb9YBYEym3fuvJB4fR1L2GPTpWPbESbVgAXHpa64): protocol: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection Nov 14 02:26:04.364912240 (114h22m2.916382711s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364936345 (114h22m2.916406806s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YdSTc2X61baRotZGev4q25BFkEspbienJtMcpbQvGgbyCRYfx): protocol: read tcp 188.166.225.170:48200->145.79.227.37:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 02:26:04.364944490 (114h22m2.916414941s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365027369 (114h22m2.91649783s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv): protocol: expected piece hash; context canceled; EOF Nov 14 02:26:04.365037989 (114h22m2.91650845s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365060712 (114h22m2.916531173s) [seg_pos={0 0}, upload=100627]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 02:26:04.365180710 (114h22m2.916651172s) [upload=100627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 02:26:04.365188695 (114h22m2.916659156s) [upload=100627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 02:26:04.675897038 (114h22m3.227367509s) [upload=100627]: Done uploading segments... Nov 14 02:26:04.675924961 (114h22m3.227395423s) [upload=100627]: Upload finished. err: <nil> Duration: 5.885330986s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [...X...........] duration: 85.129875ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/496c88a2aec2b412 FAIL (2025-11-14 02:49:37 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:54352->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 85.155544ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.74224827s trace: https://jaeger.dev.storj.io/trace/4d67fad9e2c6ec76 FAIL (2025-11-14 02:49:33 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.881728976 (114h21m4.433199447s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882337685 (114h21m4.433808146s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882761321 (114h21m4.434231782s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883755024 (114h21m4.435225485s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883875503 (114h21m4.435345974s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.884804482 (114h21m4.436274943s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885204404 (114h21m4.436674865s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885789247 (114h21m4.437259708s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886160905 (114h21m4.437631376s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886974594 (114h21m4.438445055s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887182179 (114h21m4.438652651s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887657485 (114h21m4.439127946s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888130356 (114h21m4.439600817s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888691123 (114h21m4.440161594s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:07.255882067 (114h21m5.807352528s) [upload=100627]: Next returned nil segment Nov 14 02:25:07.255900562 (114h21m5.807371023s) [upload=100627]: Waiting for error group managing segments... Nov 14 02:25:07.629575566 (114h21m6.181046027s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.802313484 (114h21m6.353783945s) [noise=true , address=64.49.13.140:4510 , piece_id=WRZE6WU5UZ56L22NNBV3GVUG3HC52W7AJFP6ULZSLQ25V76JEEQQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.903131683 (114h21m6.454602145s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.907810763 (114h21m6.459281224s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.915612680 (114h21m6.467083141s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.919665296 (114h21m6.471135757s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.927705556 (114h21m6.479176018s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937100136 (114h21m6.488570608s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937995852 (114h21m6.489466323s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.938767291 (114h21m6.490237752s) [noise=true , address=77.90.31.18:4809 , piece_id=FS5BSPEWP734H6KZQ3ILIC7BU532CJ2NHM2K3XL3G5CTM5G26TOA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.975450604 (114h21m6.526921065s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:08.060141452 (114h21m6.611611913s) [noise=true , address=77.90.31.11:4572 , piece_id=LNPRUL5YLDXS3KXUGO72RVPXYHV43Y3PMZZ4LIE3WX2SQW6VHRNA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:09.481028605 (114h21m8.032499066s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364334249 (114h22m2.91580471s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364362523 (114h22m2.915832984s) [seg_pos={0 0}, upload=100627]: Segment reached optimal threshold of 14 pieces. Nov 14 02:26:04.364726596 (114h22m2.916197057s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection Nov 14 02:26:04.364736675 (114h22m2.916207136s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364900176 (114h22m2.916370647s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BZnnbpp63tb9YBYEym3fuvJB4fR1L2GPTpWPbESbVgAXHpa64): protocol: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection Nov 14 02:26:04.364912240 (114h22m2.916382711s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364936345 (114h22m2.916406806s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YdSTc2X61baRotZGev4q25BFkEspbienJtMcpbQvGgbyCRYfx): protocol: read tcp 188.166.225.170:48200->145.79.227.37:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 02:26:04.364944490 (114h22m2.916414941s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365027369 (114h22m2.91649783s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv): protocol: expected piece hash; context canceled; EOF Nov 14 02:26:04.365037989 (114h22m2.91650845s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365060712 (114h22m2.916531173s) [seg_pos={0 0}, upload=100627]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 02:26:04.365180710 (114h22m2.916651172s) [upload=100627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 02:26:04.365188695 (114h22m2.916659156s) [upload=100627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 02:26:04.675897038 (114h22m3.227367509s) [upload=100627]: Done uploading segments... Nov 14 02:26:04.675924961 (114h22m3.227395423s) [upload=100627]: Upload finished. err: <nil> Duration: 5.742277995s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..XX........................................................] duration: 6.247348423s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:49:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.247379994s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [....X..........] duration: 77.489877ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/496c88a2aec2b412 FAIL (2025-11-14 02:48:47 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:54352->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 77.522348ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.764102141s trace: https://jaeger.dev.storj.io/trace/4d67fad9e2c6ec76 FAIL (2025-11-14 02:48:31 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.881728976 (114h21m4.433199447s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882337685 (114h21m4.433808146s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882761321 (114h21m4.434231782s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883755024 (114h21m4.435225485s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883875503 (114h21m4.435345974s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.884804482 (114h21m4.436274943s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885204404 (114h21m4.436674865s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885789247 (114h21m4.437259708s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886160905 (114h21m4.437631376s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886974594 (114h21m4.438445055s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887182179 (114h21m4.438652651s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887657485 (114h21m4.439127946s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888130356 (114h21m4.439600817s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888691123 (114h21m4.440161594s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:07.255882067 (114h21m5.807352528s) [upload=100627]: Next returned nil segment Nov 14 02:25:07.255900562 (114h21m5.807371023s) [upload=100627]: Waiting for error group managing segments... Nov 14 02:25:07.629575566 (114h21m6.181046027s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.802313484 (114h21m6.353783945s) [noise=true , address=64.49.13.140:4510 , piece_id=WRZE6WU5UZ56L22NNBV3GVUG3HC52W7AJFP6ULZSLQ25V76JEEQQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.903131683 (114h21m6.454602145s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.907810763 (114h21m6.459281224s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.915612680 (114h21m6.467083141s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.919665296 (114h21m6.471135757s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.927705556 (114h21m6.479176018s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937100136 (114h21m6.488570608s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937995852 (114h21m6.489466323s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.938767291 (114h21m6.490237752s) [noise=true , address=77.90.31.18:4809 , piece_id=FS5BSPEWP734H6KZQ3ILIC7BU532CJ2NHM2K3XL3G5CTM5G26TOA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.975450604 (114h21m6.526921065s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:08.060141452 (114h21m6.611611913s) [noise=true , address=77.90.31.11:4572 , piece_id=LNPRUL5YLDXS3KXUGO72RVPXYHV43Y3PMZZ4LIE3WX2SQW6VHRNA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:09.481028605 (114h21m8.032499066s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364334249 (114h22m2.91580471s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364362523 (114h22m2.915832984s) [seg_pos={0 0}, upload=100627]: Segment reached optimal threshold of 14 pieces. Nov 14 02:26:04.364726596 (114h22m2.916197057s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection Nov 14 02:26:04.364736675 (114h22m2.916207136s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364900176 (114h22m2.916370647s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BZnnbpp63tb9YBYEym3fuvJB4fR1L2GPTpWPbESbVgAXHpa64): protocol: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection Nov 14 02:26:04.364912240 (114h22m2.916382711s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364936345 (114h22m2.916406806s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YdSTc2X61baRotZGev4q25BFkEspbienJtMcpbQvGgbyCRYfx): protocol: read tcp 188.166.225.170:48200->145.79.227.37:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 02:26:04.364944490 (114h22m2.916414941s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365027369 (114h22m2.91649783s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv): protocol: expected piece hash; context canceled; EOF Nov 14 02:26:04.365037989 (114h22m2.91650845s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365060712 (114h22m2.916531173s) [seg_pos={0 0}, upload=100627]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 02:26:04.365180710 (114h22m2.916651172s) [upload=100627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 02:26:04.365188695 (114h22m2.916659156s) [upload=100627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 02:26:04.675897038 (114h22m3.227367509s) [upload=100627]: Done uploading segments... Nov 14 02:26:04.675924961 (114h22m3.227395423s) [upload=100627]: Upload finished. err: <nil> Duration: 5.764134403s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...XX.......................................................] duration: 5.957493076s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:48:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.957513835s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.....X.........] duration: 58.751855ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/496c88a2aec2b412 FAIL (2025-11-14 02:47:54 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:54352->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 58.775329ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....XX......................................................] duration: 5.916163438s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:47:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.916196321s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [......X........] duration: 68.438131ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/496c88a2aec2b412 FAIL (2025-11-14 02:47:21 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:54352->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 68.471285ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.918078173s trace: https://jaeger.dev.storj.io/trace/4d67fad9e2c6ec76 FAIL (2025-11-14 02:47:19 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.881728976 (114h21m4.433199447s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882337685 (114h21m4.433808146s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882761321 (114h21m4.434231782s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883755024 (114h21m4.435225485s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883875503 (114h21m4.435345974s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.884804482 (114h21m4.436274943s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885204404 (114h21m4.436674865s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885789247 (114h21m4.437259708s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886160905 (114h21m4.437631376s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886974594 (114h21m4.438445055s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887182179 (114h21m4.438652651s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887657485 (114h21m4.439127946s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888130356 (114h21m4.439600817s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888691123 (114h21m4.440161594s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:07.255882067 (114h21m5.807352528s) [upload=100627]: Next returned nil segment Nov 14 02:25:07.255900562 (114h21m5.807371023s) [upload=100627]: Waiting for error group managing segments... Nov 14 02:25:07.629575566 (114h21m6.181046027s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.802313484 (114h21m6.353783945s) [noise=true , address=64.49.13.140:4510 , piece_id=WRZE6WU5UZ56L22NNBV3GVUG3HC52W7AJFP6ULZSLQ25V76JEEQQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.903131683 (114h21m6.454602145s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.907810763 (114h21m6.459281224s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.915612680 (114h21m6.467083141s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.919665296 (114h21m6.471135757s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.927705556 (114h21m6.479176018s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937100136 (114h21m6.488570608s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937995852 (114h21m6.489466323s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.938767291 (114h21m6.490237752s) [noise=true , address=77.90.31.18:4809 , piece_id=FS5BSPEWP734H6KZQ3ILIC7BU532CJ2NHM2K3XL3G5CTM5G26TOA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.975450604 (114h21m6.526921065s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:08.060141452 (114h21m6.611611913s) [noise=true , address=77.90.31.11:4572 , piece_id=LNPRUL5YLDXS3KXUGO72RVPXYHV43Y3PMZZ4LIE3WX2SQW6VHRNA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:09.481028605 (114h21m8.032499066s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364334249 (114h22m2.91580471s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364362523 (114h22m2.915832984s) [seg_pos={0 0}, upload=100627]: Segment reached optimal threshold of 14 pieces. Nov 14 02:26:04.364726596 (114h22m2.916197057s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection Nov 14 02:26:04.364736675 (114h22m2.916207136s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364900176 (114h22m2.916370647s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BZnnbpp63tb9YBYEym3fuvJB4fR1L2GPTpWPbESbVgAXHpa64): protocol: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection Nov 14 02:26:04.364912240 (114h22m2.916382711s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364936345 (114h22m2.916406806s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YdSTc2X61baRotZGev4q25BFkEspbienJtMcpbQvGgbyCRYfx): protocol: read tcp 188.166.225.170:48200->145.79.227.37:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 02:26:04.364944490 (114h22m2.916414941s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365027369 (114h22m2.91649783s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv): protocol: expected piece hash; context canceled; EOF Nov 14 02:26:04.365037989 (114h22m2.91650845s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365060712 (114h22m2.916531173s) [seg_pos={0 0}, upload=100627]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 02:26:04.365180710 (114h22m2.916651172s) [upload=100627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 02:26:04.365188695 (114h22m2.916659156s) [upload=100627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 02:26:04.675897038 (114h22m3.227367509s) [upload=100627]: Done uploading segments... Nov 14 02:26:04.675924961 (114h22m3.227395423s) [upload=100627]: Upload finished. err: <nil> Duration: 5.918107979s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.738665726s trace: https://jaeger.dev.storj.io/trace/4d67fad9e2c6ec76 FAIL (2025-11-14 02:46:38 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.881728976 (114h21m4.433199447s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882337685 (114h21m4.433808146s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882761321 (114h21m4.434231782s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883755024 (114h21m4.435225485s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883875503 (114h21m4.435345974s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.884804482 (114h21m4.436274943s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885204404 (114h21m4.436674865s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885789247 (114h21m4.437259708s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886160905 (114h21m4.437631376s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886974594 (114h21m4.438445055s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887182179 (114h21m4.438652651s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887657485 (114h21m4.439127946s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888130356 (114h21m4.439600817s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888691123 (114h21m4.440161594s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:07.255882067 (114h21m5.807352528s) [upload=100627]: Next returned nil segment Nov 14 02:25:07.255900562 (114h21m5.807371023s) [upload=100627]: Waiting for error group managing segments... Nov 14 02:25:07.629575566 (114h21m6.181046027s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.802313484 (114h21m6.353783945s) [noise=true , address=64.49.13.140:4510 , piece_id=WRZE6WU5UZ56L22NNBV3GVUG3HC52W7AJFP6ULZSLQ25V76JEEQQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.903131683 (114h21m6.454602145s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.907810763 (114h21m6.459281224s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.915612680 (114h21m6.467083141s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.919665296 (114h21m6.471135757s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.927705556 (114h21m6.479176018s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937100136 (114h21m6.488570608s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937995852 (114h21m6.489466323s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.938767291 (114h21m6.490237752s) [noise=true , address=77.90.31.18:4809 , piece_id=FS5BSPEWP734H6KZQ3ILIC7BU532CJ2NHM2K3XL3G5CTM5G26TOA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.975450604 (114h21m6.526921065s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:08.060141452 (114h21m6.611611913s) [noise=true , address=77.90.31.11:4572 , piece_id=LNPRUL5YLDXS3KXUGO72RVPXYHV43Y3PMZZ4LIE3WX2SQW6VHRNA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:09.481028605 (114h21m8.032499066s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364334249 (114h22m2.91580471s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364362523 (114h22m2.915832984s) [seg_pos={0 0}, upload=100627]: Segment reached optimal threshold of 14 pieces. Nov 14 02:26:04.364726596 (114h22m2.916197057s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection Nov 14 02:26:04.364736675 (114h22m2.916207136s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364900176 (114h22m2.916370647s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BZnnbpp63tb9YBYEym3fuvJB4fR1L2GPTpWPbESbVgAXHpa64): protocol: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection Nov 14 02:26:04.364912240 (114h22m2.916382711s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364936345 (114h22m2.916406806s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YdSTc2X61baRotZGev4q25BFkEspbienJtMcpbQvGgbyCRYfx): protocol: read tcp 188.166.225.170:48200->145.79.227.37:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 02:26:04.364944490 (114h22m2.916414941s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365027369 (114h22m2.91649783s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv): protocol: expected piece hash; context canceled; EOF Nov 14 02:26:04.365037989 (114h22m2.91650845s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365060712 (114h22m2.916531173s) [seg_pos={0 0}, upload=100627]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 02:26:04.365180710 (114h22m2.916651172s) [upload=100627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 02:26:04.365188695 (114h22m2.916659156s) [upload=100627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 02:26:04.675897038 (114h22m3.227367509s) [upload=100627]: Done uploading segments... Nov 14 02:26:04.675924961 (114h22m3.227395423s) [upload=100627]: Upload finished. err: <nil> Duration: 5.738759273s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....XX.....................................................] duration: 5.93008827s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:46:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.930119229s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.......X.......] duration: 58.799407ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/496c88a2aec2b412 FAIL (2025-11-14 02:46:27 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:54352->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 58.835766ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [........X......] duration: 56.483576ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/496c88a2aec2b412 FAIL (2025-11-14 02:45:52 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:54352->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 56.511239ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......XX....................................................] duration: 5.976177754s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:45:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.976210137s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.757791423s trace: https://jaeger.dev.storj.io/trace/4d67fad9e2c6ec76 FAIL (2025-11-14 02:45:25 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.881728976 (114h21m4.433199447s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882337685 (114h21m4.433808146s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882761321 (114h21m4.434231782s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883755024 (114h21m4.435225485s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883875503 (114h21m4.435345974s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.884804482 (114h21m4.436274943s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885204404 (114h21m4.436674865s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885789247 (114h21m4.437259708s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886160905 (114h21m4.437631376s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886974594 (114h21m4.438445055s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887182179 (114h21m4.438652651s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887657485 (114h21m4.439127946s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888130356 (114h21m4.439600817s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888691123 (114h21m4.440161594s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:07.255882067 (114h21m5.807352528s) [upload=100627]: Next returned nil segment Nov 14 02:25:07.255900562 (114h21m5.807371023s) [upload=100627]: Waiting for error group managing segments... Nov 14 02:25:07.629575566 (114h21m6.181046027s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.802313484 (114h21m6.353783945s) [noise=true , address=64.49.13.140:4510 , piece_id=WRZE6WU5UZ56L22NNBV3GVUG3HC52W7AJFP6ULZSLQ25V76JEEQQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.903131683 (114h21m6.454602145s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.907810763 (114h21m6.459281224s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.915612680 (114h21m6.467083141s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.919665296 (114h21m6.471135757s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.927705556 (114h21m6.479176018s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937100136 (114h21m6.488570608s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937995852 (114h21m6.489466323s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.938767291 (114h21m6.490237752s) [noise=true , address=77.90.31.18:4809 , piece_id=FS5BSPEWP734H6KZQ3ILIC7BU532CJ2NHM2K3XL3G5CTM5G26TOA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.975450604 (114h21m6.526921065s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:08.060141452 (114h21m6.611611913s) [noise=true , address=77.90.31.11:4572 , piece_id=LNPRUL5YLDXS3KXUGO72RVPXYHV43Y3PMZZ4LIE3WX2SQW6VHRNA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:09.481028605 (114h21m8.032499066s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364334249 (114h22m2.91580471s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364362523 (114h22m2.915832984s) [seg_pos={0 0}, upload=100627]: Segment reached optimal threshold of 14 pieces. Nov 14 02:26:04.364726596 (114h22m2.916197057s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection Nov 14 02:26:04.364736675 (114h22m2.916207136s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364900176 (114h22m2.916370647s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BZnnbpp63tb9YBYEym3fuvJB4fR1L2GPTpWPbESbVgAXHpa64): protocol: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection Nov 14 02:26:04.364912240 (114h22m2.916382711s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364936345 (114h22m2.916406806s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YdSTc2X61baRotZGev4q25BFkEspbienJtMcpbQvGgbyCRYfx): protocol: read tcp 188.166.225.170:48200->145.79.227.37:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 02:26:04.364944490 (114h22m2.916414941s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365027369 (114h22m2.91649783s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv): protocol: expected piece hash; context canceled; EOF Nov 14 02:26:04.365037989 (114h22m2.91650845s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365060712 (114h22m2.916531173s) [seg_pos={0 0}, upload=100627]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 02:26:04.365180710 (114h22m2.916651172s) [upload=100627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 02:26:04.365188695 (114h22m2.916659156s) [upload=100627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 02:26:04.675897038 (114h22m3.227367509s) [upload=100627]: Done uploading segments... Nov 14 02:26:04.675924961 (114h22m3.227395423s) [upload=100627]: Upload finished. err: <nil> Duration: 5.757822762s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......XX...................................................] duration: 6.239687764s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:45:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.239802904s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.........X.....] duration: 77.848681ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/496c88a2aec2b412 FAIL (2025-11-14 02:44:59 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:54352->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 77.88012ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.659378437s trace: https://jaeger.dev.storj.io/trace/4d67fad9e2c6ec76 FAIL (2025-11-14 02:44:46 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.881728976 (114h21m4.433199447s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882337685 (114h21m4.433808146s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882761321 (114h21m4.434231782s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883755024 (114h21m4.435225485s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883875503 (114h21m4.435345974s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.884804482 (114h21m4.436274943s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885204404 (114h21m4.436674865s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885789247 (114h21m4.437259708s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886160905 (114h21m4.437631376s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886974594 (114h21m4.438445055s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887182179 (114h21m4.438652651s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887657485 (114h21m4.439127946s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888130356 (114h21m4.439600817s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888691123 (114h21m4.440161594s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:07.255882067 (114h21m5.807352528s) [upload=100627]: Next returned nil segment Nov 14 02:25:07.255900562 (114h21m5.807371023s) [upload=100627]: Waiting for error group managing segments... Nov 14 02:25:07.629575566 (114h21m6.181046027s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.802313484 (114h21m6.353783945s) [noise=true , address=64.49.13.140:4510 , piece_id=WRZE6WU5UZ56L22NNBV3GVUG3HC52W7AJFP6ULZSLQ25V76JEEQQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.903131683 (114h21m6.454602145s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.907810763 (114h21m6.459281224s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.915612680 (114h21m6.467083141s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.919665296 (114h21m6.471135757s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.927705556 (114h21m6.479176018s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937100136 (114h21m6.488570608s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937995852 (114h21m6.489466323s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.938767291 (114h21m6.490237752s) [noise=true , address=77.90.31.18:4809 , piece_id=FS5BSPEWP734H6KZQ3ILIC7BU532CJ2NHM2K3XL3G5CTM5G26TOA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.975450604 (114h21m6.526921065s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:08.060141452 (114h21m6.611611913s) [noise=true , address=77.90.31.11:4572 , piece_id=LNPRUL5YLDXS3KXUGO72RVPXYHV43Y3PMZZ4LIE3WX2SQW6VHRNA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:09.481028605 (114h21m8.032499066s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364334249 (114h22m2.91580471s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364362523 (114h22m2.915832984s) [seg_pos={0 0}, upload=100627]: Segment reached optimal threshold of 14 pieces. Nov 14 02:26:04.364726596 (114h22m2.916197057s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection Nov 14 02:26:04.364736675 (114h22m2.916207136s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364900176 (114h22m2.916370647s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BZnnbpp63tb9YBYEym3fuvJB4fR1L2GPTpWPbESbVgAXHpa64): protocol: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection Nov 14 02:26:04.364912240 (114h22m2.916382711s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364936345 (114h22m2.916406806s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YdSTc2X61baRotZGev4q25BFkEspbienJtMcpbQvGgbyCRYfx): protocol: read tcp 188.166.225.170:48200->145.79.227.37:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 02:26:04.364944490 (114h22m2.916414941s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365027369 (114h22m2.91649783s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv): protocol: expected piece hash; context canceled; EOF Nov 14 02:26:04.365037989 (114h22m2.91650845s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365060712 (114h22m2.916531173s) [seg_pos={0 0}, upload=100627]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 02:26:04.365180710 (114h22m2.916651172s) [upload=100627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 02:26:04.365188695 (114h22m2.916659156s) [upload=100627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 02:26:04.675897038 (114h22m3.227367509s) [upload=100627]: Done uploading segments... Nov 14 02:26:04.675924961 (114h22m3.227395423s) [upload=100627]: Upload finished. err: <nil> Duration: 5.659501351s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........XX..................................................] duration: 6.047798238s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:44:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.047832533s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..........X....] duration: 70.89175ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/496c88a2aec2b412 FAIL (2025-11-14 02:44:09 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:54352->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 70.922139ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.827166638s trace: https://jaeger.dev.storj.io/trace/4d67fad9e2c6ec76 FAIL (2025-11-14 02:43:51 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.881728976 (114h21m4.433199447s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882337685 (114h21m4.433808146s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882761321 (114h21m4.434231782s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883755024 (114h21m4.435225485s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883875503 (114h21m4.435345974s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.884804482 (114h21m4.436274943s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885204404 (114h21m4.436674865s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885789247 (114h21m4.437259708s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886160905 (114h21m4.437631376s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886974594 (114h21m4.438445055s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887182179 (114h21m4.438652651s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887657485 (114h21m4.439127946s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888130356 (114h21m4.439600817s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888691123 (114h21m4.440161594s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:07.255882067 (114h21m5.807352528s) [upload=100627]: Next returned nil segment Nov 14 02:25:07.255900562 (114h21m5.807371023s) [upload=100627]: Waiting for error group managing segments... Nov 14 02:25:07.629575566 (114h21m6.181046027s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.802313484 (114h21m6.353783945s) [noise=true , address=64.49.13.140:4510 , piece_id=WRZE6WU5UZ56L22NNBV3GVUG3HC52W7AJFP6ULZSLQ25V76JEEQQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.903131683 (114h21m6.454602145s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.907810763 (114h21m6.459281224s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.915612680 (114h21m6.467083141s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.919665296 (114h21m6.471135757s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.927705556 (114h21m6.479176018s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937100136 (114h21m6.488570608s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937995852 (114h21m6.489466323s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.938767291 (114h21m6.490237752s) [noise=true , address=77.90.31.18:4809 , piece_id=FS5BSPEWP734H6KZQ3ILIC7BU532CJ2NHM2K3XL3G5CTM5G26TOA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.975450604 (114h21m6.526921065s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:08.060141452 (114h21m6.611611913s) [noise=true , address=77.90.31.11:4572 , piece_id=LNPRUL5YLDXS3KXUGO72RVPXYHV43Y3PMZZ4LIE3WX2SQW6VHRNA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:09.481028605 (114h21m8.032499066s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364334249 (114h22m2.91580471s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364362523 (114h22m2.915832984s) [seg_pos={0 0}, upload=100627]: Segment reached optimal threshold of 14 pieces. Nov 14 02:26:04.364726596 (114h22m2.916197057s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection Nov 14 02:26:04.364736675 (114h22m2.916207136s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364900176 (114h22m2.916370647s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BZnnbpp63tb9YBYEym3fuvJB4fR1L2GPTpWPbESbVgAXHpa64): protocol: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection Nov 14 02:26:04.364912240 (114h22m2.916382711s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364936345 (114h22m2.916406806s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YdSTc2X61baRotZGev4q25BFkEspbienJtMcpbQvGgbyCRYfx): protocol: read tcp 188.166.225.170:48200->145.79.227.37:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 02:26:04.364944490 (114h22m2.916414941s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365027369 (114h22m2.91649783s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv): protocol: expected piece hash; context canceled; EOF Nov 14 02:26:04.365037989 (114h22m2.91650845s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365060712 (114h22m2.916531173s) [seg_pos={0 0}, upload=100627]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 02:26:04.365180710 (114h22m2.916651172s) [upload=100627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 02:26:04.365188695 (114h22m2.916659156s) [upload=100627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 02:26:04.675897038 (114h22m3.227367509s) [upload=100627]: Done uploading segments... Nov 14 02:26:04.675924961 (114h22m3.227395423s) [upload=100627]: Upload finished. err: <nil> Duration: 5.827226191s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........XX.................................................] duration: 5.996660339s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:43:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.996689272s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [...........X...] duration: 69.299499ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/496c88a2aec2b412 FAIL (2025-11-14 02:43:23 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:54352->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 69.319339ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.726735876s trace: https://jaeger.dev.storj.io/trace/4d67fad9e2c6ec76 FAIL (2025-11-14 02:43:21 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.881728976 (114h21m4.433199447s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882337685 (114h21m4.433808146s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882761321 (114h21m4.434231782s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883755024 (114h21m4.435225485s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883875503 (114h21m4.435345974s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.884804482 (114h21m4.436274943s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885204404 (114h21m4.436674865s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885789247 (114h21m4.437259708s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886160905 (114h21m4.437631376s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886974594 (114h21m4.438445055s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887182179 (114h21m4.438652651s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887657485 (114h21m4.439127946s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888130356 (114h21m4.439600817s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888691123 (114h21m4.440161594s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:07.255882067 (114h21m5.807352528s) [upload=100627]: Next returned nil segment Nov 14 02:25:07.255900562 (114h21m5.807371023s) [upload=100627]: Waiting for error group managing segments... Nov 14 02:25:07.629575566 (114h21m6.181046027s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.802313484 (114h21m6.353783945s) [noise=true , address=64.49.13.140:4510 , piece_id=WRZE6WU5UZ56L22NNBV3GVUG3HC52W7AJFP6ULZSLQ25V76JEEQQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.903131683 (114h21m6.454602145s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.907810763 (114h21m6.459281224s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.915612680 (114h21m6.467083141s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.919665296 (114h21m6.471135757s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.927705556 (114h21m6.479176018s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937100136 (114h21m6.488570608s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937995852 (114h21m6.489466323s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.938767291 (114h21m6.490237752s) [noise=true , address=77.90.31.18:4809 , piece_id=FS5BSPEWP734H6KZQ3ILIC7BU532CJ2NHM2K3XL3G5CTM5G26TOA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.975450604 (114h21m6.526921065s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:08.060141452 (114h21m6.611611913s) [noise=true , address=77.90.31.11:4572 , piece_id=LNPRUL5YLDXS3KXUGO72RVPXYHV43Y3PMZZ4LIE3WX2SQW6VHRNA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:09.481028605 (114h21m8.032499066s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364334249 (114h22m2.91580471s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364362523 (114h22m2.915832984s) [seg_pos={0 0}, upload=100627]: Segment reached optimal threshold of 14 pieces. Nov 14 02:26:04.364726596 (114h22m2.916197057s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection Nov 14 02:26:04.364736675 (114h22m2.916207136s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364900176 (114h22m2.916370647s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BZnnbpp63tb9YBYEym3fuvJB4fR1L2GPTpWPbESbVgAXHpa64): protocol: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection Nov 14 02:26:04.364912240 (114h22m2.916382711s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364936345 (114h22m2.916406806s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YdSTc2X61baRotZGev4q25BFkEspbienJtMcpbQvGgbyCRYfx): protocol: read tcp 188.166.225.170:48200->145.79.227.37:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 02:26:04.364944490 (114h22m2.916414941s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365027369 (114h22m2.91649783s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv): protocol: expected piece hash; context canceled; EOF Nov 14 02:26:04.365037989 (114h22m2.91650845s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365060712 (114h22m2.916531173s) [seg_pos={0 0}, upload=100627]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 02:26:04.365180710 (114h22m2.916651172s) [upload=100627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 02:26:04.365188695 (114h22m2.916659156s) [upload=100627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 02:26:04.675897038 (114h22m3.227367509s) [upload=100627]: Done uploading segments... Nov 14 02:26:04.675924961 (114h22m3.227395423s) [upload=100627]: Upload finished. err: <nil> Duration: 5.726767397s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........XX................................................] duration: 6.10532462s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:43:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.105356039s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.63019216s trace: https://jaeger.dev.storj.io/trace/4d67fad9e2c6ec76 FAIL (2025-11-14 02:42:40 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.881728976 (114h21m4.433199447s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882337685 (114h21m4.433808146s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882761321 (114h21m4.434231782s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883755024 (114h21m4.435225485s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883875503 (114h21m4.435345974s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.884804482 (114h21m4.436274943s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885204404 (114h21m4.436674865s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885789247 (114h21m4.437259708s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886160905 (114h21m4.437631376s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886974594 (114h21m4.438445055s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887182179 (114h21m4.438652651s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887657485 (114h21m4.439127946s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888130356 (114h21m4.439600817s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888691123 (114h21m4.440161594s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:07.255882067 (114h21m5.807352528s) [upload=100627]: Next returned nil segment Nov 14 02:25:07.255900562 (114h21m5.807371023s) [upload=100627]: Waiting for error group managing segments... Nov 14 02:25:07.629575566 (114h21m6.181046027s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.802313484 (114h21m6.353783945s) [noise=true , address=64.49.13.140:4510 , piece_id=WRZE6WU5UZ56L22NNBV3GVUG3HC52W7AJFP6ULZSLQ25V76JEEQQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.903131683 (114h21m6.454602145s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.907810763 (114h21m6.459281224s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.915612680 (114h21m6.467083141s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.919665296 (114h21m6.471135757s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.927705556 (114h21m6.479176018s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937100136 (114h21m6.488570608s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937995852 (114h21m6.489466323s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.938767291 (114h21m6.490237752s) [noise=true , address=77.90.31.18:4809 , piece_id=FS5BSPEWP734H6KZQ3ILIC7BU532CJ2NHM2K3XL3G5CTM5G26TOA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.975450604 (114h21m6.526921065s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:08.060141452 (114h21m6.611611913s) [noise=true , address=77.90.31.11:4572 , piece_id=LNPRUL5YLDXS3KXUGO72RVPXYHV43Y3PMZZ4LIE3WX2SQW6VHRNA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:09.481028605 (114h21m8.032499066s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364334249 (114h22m2.91580471s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364362523 (114h22m2.915832984s) [seg_pos={0 0}, upload=100627]: Segment reached optimal threshold of 14 pieces. Nov 14 02:26:04.364726596 (114h22m2.916197057s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection Nov 14 02:26:04.364736675 (114h22m2.916207136s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364900176 (114h22m2.916370647s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BZnnbpp63tb9YBYEym3fuvJB4fR1L2GPTpWPbESbVgAXHpa64): protocol: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection Nov 14 02:26:04.364912240 (114h22m2.916382711s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364936345 (114h22m2.916406806s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YdSTc2X61baRotZGev4q25BFkEspbienJtMcpbQvGgbyCRYfx): protocol: read tcp 188.166.225.170:48200->145.79.227.37:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 02:26:04.364944490 (114h22m2.916414941s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365027369 (114h22m2.91649783s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv): protocol: expected piece hash; context canceled; EOF Nov 14 02:26:04.365037989 (114h22m2.91650845s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365060712 (114h22m2.916531173s) [seg_pos={0 0}, upload=100627]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 02:26:04.365180710 (114h22m2.916651172s) [upload=100627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 02:26:04.365188695 (114h22m2.916659156s) [upload=100627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 02:26:04.675897038 (114h22m3.227367509s) [upload=100627]: Done uploading segments... Nov 14 02:26:04.675924961 (114h22m3.227395423s) [upload=100627]: Upload finished. err: <nil> Duration: 5.630219762s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [............X..] duration: 69.98183ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/496c88a2aec2b412 FAIL (2025-11-14 02:42:12 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:54352->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 70.006677ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........XX...............................................] duration: 5.933899717s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:41:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.933921999s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.811035869s trace: https://jaeger.dev.storj.io/trace/4d67fad9e2c6ec76 FAIL (2025-11-14 02:41:51 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.881728976 (114h21m4.433199447s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882337685 (114h21m4.433808146s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882761321 (114h21m4.434231782s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883755024 (114h21m4.435225485s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883875503 (114h21m4.435345974s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.884804482 (114h21m4.436274943s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885204404 (114h21m4.436674865s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885789247 (114h21m4.437259708s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886160905 (114h21m4.437631376s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886974594 (114h21m4.438445055s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887182179 (114h21m4.438652651s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887657485 (114h21m4.439127946s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888130356 (114h21m4.439600817s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888691123 (114h21m4.440161594s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:07.255882067 (114h21m5.807352528s) [upload=100627]: Next returned nil segment Nov 14 02:25:07.255900562 (114h21m5.807371023s) [upload=100627]: Waiting for error group managing segments... Nov 14 02:25:07.629575566 (114h21m6.181046027s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.802313484 (114h21m6.353783945s) [noise=true , address=64.49.13.140:4510 , piece_id=WRZE6WU5UZ56L22NNBV3GVUG3HC52W7AJFP6ULZSLQ25V76JEEQQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.903131683 (114h21m6.454602145s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.907810763 (114h21m6.459281224s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.915612680 (114h21m6.467083141s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.919665296 (114h21m6.471135757s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.927705556 (114h21m6.479176018s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937100136 (114h21m6.488570608s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937995852 (114h21m6.489466323s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.938767291 (114h21m6.490237752s) [noise=true , address=77.90.31.18:4809 , piece_id=FS5BSPEWP734H6KZQ3ILIC7BU532CJ2NHM2K3XL3G5CTM5G26TOA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.975450604 (114h21m6.526921065s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:08.060141452 (114h21m6.611611913s) [noise=true , address=77.90.31.11:4572 , piece_id=LNPRUL5YLDXS3KXUGO72RVPXYHV43Y3PMZZ4LIE3WX2SQW6VHRNA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:09.481028605 (114h21m8.032499066s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364334249 (114h22m2.91580471s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364362523 (114h22m2.915832984s) [seg_pos={0 0}, upload=100627]: Segment reached optimal threshold of 14 pieces. Nov 14 02:26:04.364726596 (114h22m2.916197057s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection Nov 14 02:26:04.364736675 (114h22m2.916207136s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364900176 (114h22m2.916370647s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BZnnbpp63tb9YBYEym3fuvJB4fR1L2GPTpWPbESbVgAXHpa64): protocol: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection Nov 14 02:26:04.364912240 (114h22m2.916382711s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364936345 (114h22m2.916406806s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YdSTc2X61baRotZGev4q25BFkEspbienJtMcpbQvGgbyCRYfx): protocol: read tcp 188.166.225.170:48200->145.79.227.37:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 02:26:04.364944490 (114h22m2.916414941s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365027369 (114h22m2.91649783s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv): protocol: expected piece hash; context canceled; EOF Nov 14 02:26:04.365037989 (114h22m2.91650845s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365060712 (114h22m2.916531173s) [seg_pos={0 0}, upload=100627]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 02:26:04.365180710 (114h22m2.916651172s) [upload=100627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 02:26:04.365188695 (114h22m2.916659156s) [upload=100627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 02:26:04.675897038 (114h22m3.227367509s) [upload=100627]: Done uploading segments... Nov 14 02:26:04.675924961 (114h22m3.227395423s) [upload=100627]: Upload finished. err: <nil> Duration: 5.811056909s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.............X.] duration: 61.143991ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/496c88a2aec2b412 FAIL (2025-11-14 02:41:46 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:54352->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 61.171834ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..............X] duration: 439.383254ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/496c88a2aec2b412 FAIL (2025-11-14 02:41:08 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:54352->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 439.414613ms core website

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)