Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 3.503657025s trace: https://jaeger.dev.storj.io/trace/4bb85e67f0d4ec1d FAIL (2025-12-27 23:13:00 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.50368114s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X...X.....X...X..X....X.......X.....X.......................] duration: 6.124863601s trace: https://jaeger.dev.storj.io/trace/63027bab39a48860 FAIL (2025-12-27 23:12:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA) context deadline exceeded ======LOGS====== Dec 27 22:55:03.533152286 (535h26m3.404235331s) [upload=469598]: Starting upload Dec 27 22:55:03.533478477 (535h26m3.404561522s) [upload=469598]: Done starting upload Dec 27 22:55:03.533881645 (535h26m3.40496469s) [upload=469598]: Uploading segments... Dec 27 22:55:03.544825083 (535h26m3.415908128s) [upload=469598]: Got next segment. Inline: false Dec 27 22:55:03.544990097 (535h26m3.416073142s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:55:03.544999244 (535h26m3.41608229s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:55:03.791379841 (535h26m3.662462886s) [seg_pos={0 0}, upload=469598]: Begin upload segment... Dec 27 22:55:03.791445777 (535h26m3.662528812s) [seg_pos={0 0}, upload=469598]: Done begin upload segment. Dec 27 22:55:03.791485131 (535h26m3.662568176s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.791591594 (535h26m3.662674639s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.793523884 (535h26m3.664606939s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.794185644 (535h26m3.665268689s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.795222378 (535h26m3.666305434s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797183563 (535h26m3.668266618s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797315324 (535h26m3.668398379s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.799339678 (535h26m3.670422733s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.800966926 (535h26m3.672049981s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.802493253 (535h26m3.673576298s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.819392662 (535h26m3.690475708s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.823879014 (535h26m3.694962069s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.828981661 (535h26m3.700064716s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.830805173 (535h26m3.701888238s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.832266125 (535h26m3.70334918s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.836751787 (535h26m3.707834832s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:05.507079228 (535h26m5.378162263s) [upload=469598]: Next returned nil segment Dec 27 22:55:05.507093797 (535h26m5.378176842s) [upload=469598]: Waiting for error group managing segments... Dec 27 22:55:05.795763839 (535h26m5.666846884s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.802905938 (535h26m5.673988983s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.803078106 (535h26m5.674161151s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.812284336 (535h26m5.683367381s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.829206028 (535h26m5.700289073s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.835000362 (535h26m5.706083397s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.836389036 (535h26m5.707472071s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.843026765 (535h26m5.7141098s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.859561069 (535h26m5.730644104s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.871635229 (535h26m5.742718274s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.080787678 (535h26m5.951870724s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.094740707 (535h26m5.965823752s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122401899 (535h26m5.993484944s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122423880 (535h26m5.993506925s) [seg_pos={0 0}, upload=469598]: Segment reached optimal threshold of 13 pieces. Dec 27 22:55:06.122791581 (535h26m5.993874616s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection Dec 27 22:55:06.122800467 (535h26m5.993883502s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.122806378 (535h26m5.993889413s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.122825335 (535h26m5.99390838s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123315488 (535h26m5.994398523s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.123323623 (535h26m5.994406668s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123341488 (535h26m5.994424533s) [seg_pos={0 0}, upload=469598]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:55:06.123451447 (535h26m5.994534492s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:55:06.123459001 (535h26m5.994542046s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:55:06.413593572 (535h26m6.284676607s) [upload=469598]: Done uploading segments... Dec 27 22:55:06.413613590 (535h26m6.284696626s) [upload=469598]: Upload finished. err: <nil> Duration: 6.124893859s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 3.563046923s trace: https://jaeger.dev.storj.io/trace/4bb85e67f0d4ec1d FAIL (2025-12-27 23:11:49 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.563073884s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X...X.....X...X..X....X.......X.....X......................] duration: 18.369895775s trace: https://jaeger.dev.storj.io/trace/63027bab39a48860 FAIL (2025-12-27 23:11:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA) context deadline exceeded ======LOGS====== Dec 27 22:55:03.533152286 (535h26m3.404235331s) [upload=469598]: Starting upload Dec 27 22:55:03.533478477 (535h26m3.404561522s) [upload=469598]: Done starting upload Dec 27 22:55:03.533881645 (535h26m3.40496469s) [upload=469598]: Uploading segments... Dec 27 22:55:03.544825083 (535h26m3.415908128s) [upload=469598]: Got next segment. Inline: false Dec 27 22:55:03.544990097 (535h26m3.416073142s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:55:03.544999244 (535h26m3.41608229s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:55:03.791379841 (535h26m3.662462886s) [seg_pos={0 0}, upload=469598]: Begin upload segment... Dec 27 22:55:03.791445777 (535h26m3.662528812s) [seg_pos={0 0}, upload=469598]: Done begin upload segment. Dec 27 22:55:03.791485131 (535h26m3.662568176s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.791591594 (535h26m3.662674639s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.793523884 (535h26m3.664606939s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.794185644 (535h26m3.665268689s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.795222378 (535h26m3.666305434s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797183563 (535h26m3.668266618s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797315324 (535h26m3.668398379s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.799339678 (535h26m3.670422733s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.800966926 (535h26m3.672049981s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.802493253 (535h26m3.673576298s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.819392662 (535h26m3.690475708s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.823879014 (535h26m3.694962069s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.828981661 (535h26m3.700064716s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.830805173 (535h26m3.701888238s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.832266125 (535h26m3.70334918s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.836751787 (535h26m3.707834832s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:05.507079228 (535h26m5.378162263s) [upload=469598]: Next returned nil segment Dec 27 22:55:05.507093797 (535h26m5.378176842s) [upload=469598]: Waiting for error group managing segments... Dec 27 22:55:05.795763839 (535h26m5.666846884s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.802905938 (535h26m5.673988983s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.803078106 (535h26m5.674161151s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.812284336 (535h26m5.683367381s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.829206028 (535h26m5.700289073s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.835000362 (535h26m5.706083397s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.836389036 (535h26m5.707472071s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.843026765 (535h26m5.7141098s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.859561069 (535h26m5.730644104s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.871635229 (535h26m5.742718274s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.080787678 (535h26m5.951870724s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.094740707 (535h26m5.965823752s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122401899 (535h26m5.993484944s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122423880 (535h26m5.993506925s) [seg_pos={0 0}, upload=469598]: Segment reached optimal threshold of 13 pieces. Dec 27 22:55:06.122791581 (535h26m5.993874616s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection Dec 27 22:55:06.122800467 (535h26m5.993883502s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.122806378 (535h26m5.993889413s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.122825335 (535h26m5.99390838s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123315488 (535h26m5.994398523s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.123323623 (535h26m5.994406668s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123341488 (535h26m5.994424533s) [seg_pos={0 0}, upload=469598]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:55:06.123451447 (535h26m5.994534492s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:55:06.123459001 (535h26m5.994542046s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:55:06.413593572 (535h26m6.284676607s) [upload=469598]: Done uploading segments... Dec 27 22:55:06.413613590 (535h26m6.284696626s) [upload=469598]: Upload finished. err: <nil> Duration: 18.369913709s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 3.582529541s trace: https://jaeger.dev.storj.io/trace/4bb85e67f0d4ec1d FAIL (2025-12-27 23:11:21 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.582558156s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X.X...X.....X...X..X....X.......X.....X.....................] duration: 7.135703765s trace: https://jaeger.dev.storj.io/trace/63027bab39a48860 FAIL (2025-12-27 23:11:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA) context deadline exceeded ======LOGS====== Dec 27 22:55:03.533152286 (535h26m3.404235331s) [upload=469598]: Starting upload Dec 27 22:55:03.533478477 (535h26m3.404561522s) [upload=469598]: Done starting upload Dec 27 22:55:03.533881645 (535h26m3.40496469s) [upload=469598]: Uploading segments... Dec 27 22:55:03.544825083 (535h26m3.415908128s) [upload=469598]: Got next segment. Inline: false Dec 27 22:55:03.544990097 (535h26m3.416073142s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:55:03.544999244 (535h26m3.41608229s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:55:03.791379841 (535h26m3.662462886s) [seg_pos={0 0}, upload=469598]: Begin upload segment... Dec 27 22:55:03.791445777 (535h26m3.662528812s) [seg_pos={0 0}, upload=469598]: Done begin upload segment. Dec 27 22:55:03.791485131 (535h26m3.662568176s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.791591594 (535h26m3.662674639s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.793523884 (535h26m3.664606939s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.794185644 (535h26m3.665268689s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.795222378 (535h26m3.666305434s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797183563 (535h26m3.668266618s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797315324 (535h26m3.668398379s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.799339678 (535h26m3.670422733s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.800966926 (535h26m3.672049981s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.802493253 (535h26m3.673576298s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.819392662 (535h26m3.690475708s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.823879014 (535h26m3.694962069s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.828981661 (535h26m3.700064716s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.830805173 (535h26m3.701888238s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.832266125 (535h26m3.70334918s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.836751787 (535h26m3.707834832s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:05.507079228 (535h26m5.378162263s) [upload=469598]: Next returned nil segment Dec 27 22:55:05.507093797 (535h26m5.378176842s) [upload=469598]: Waiting for error group managing segments... Dec 27 22:55:05.795763839 (535h26m5.666846884s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.802905938 (535h26m5.673988983s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.803078106 (535h26m5.674161151s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.812284336 (535h26m5.683367381s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.829206028 (535h26m5.700289073s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.835000362 (535h26m5.706083397s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.836389036 (535h26m5.707472071s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.843026765 (535h26m5.7141098s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.859561069 (535h26m5.730644104s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.871635229 (535h26m5.742718274s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.080787678 (535h26m5.951870724s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.094740707 (535h26m5.965823752s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122401899 (535h26m5.993484944s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122423880 (535h26m5.993506925s) [seg_pos={0 0}, upload=469598]: Segment reached optimal threshold of 13 pieces. Dec 27 22:55:06.122791581 (535h26m5.993874616s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection Dec 27 22:55:06.122800467 (535h26m5.993883502s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.122806378 (535h26m5.993889413s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.122825335 (535h26m5.99390838s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123315488 (535h26m5.994398523s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.123323623 (535h26m5.994406668s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123341488 (535h26m5.994424533s) [seg_pos={0 0}, upload=469598]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:55:06.123451447 (535h26m5.994534492s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:55:06.123459001 (535h26m5.994542046s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:55:06.413593572 (535h26m6.284676607s) [upload=469598]: Done uploading segments... Dec 27 22:55:06.413613590 (535h26m6.284696626s) [upload=469598]: Upload finished. err: <nil> Duration: 7.135721008s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X.X...X.....X...X..X....X.......X.....X....................] duration: 5.922091951s trace: https://jaeger.dev.storj.io/trace/63027bab39a48860 FAIL (2025-12-27 23:10:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA) context deadline exceeded ======LOGS====== Dec 27 22:55:03.533152286 (535h26m3.404235331s) [upload=469598]: Starting upload Dec 27 22:55:03.533478477 (535h26m3.404561522s) [upload=469598]: Done starting upload Dec 27 22:55:03.533881645 (535h26m3.40496469s) [upload=469598]: Uploading segments... Dec 27 22:55:03.544825083 (535h26m3.415908128s) [upload=469598]: Got next segment. Inline: false Dec 27 22:55:03.544990097 (535h26m3.416073142s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:55:03.544999244 (535h26m3.41608229s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:55:03.791379841 (535h26m3.662462886s) [seg_pos={0 0}, upload=469598]: Begin upload segment... Dec 27 22:55:03.791445777 (535h26m3.662528812s) [seg_pos={0 0}, upload=469598]: Done begin upload segment. Dec 27 22:55:03.791485131 (535h26m3.662568176s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.791591594 (535h26m3.662674639s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.793523884 (535h26m3.664606939s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.794185644 (535h26m3.665268689s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.795222378 (535h26m3.666305434s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797183563 (535h26m3.668266618s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797315324 (535h26m3.668398379s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.799339678 (535h26m3.670422733s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.800966926 (535h26m3.672049981s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.802493253 (535h26m3.673576298s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.819392662 (535h26m3.690475708s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.823879014 (535h26m3.694962069s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.828981661 (535h26m3.700064716s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.830805173 (535h26m3.701888238s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.832266125 (535h26m3.70334918s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.836751787 (535h26m3.707834832s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:05.507079228 (535h26m5.378162263s) [upload=469598]: Next returned nil segment Dec 27 22:55:05.507093797 (535h26m5.378176842s) [upload=469598]: Waiting for error group managing segments... Dec 27 22:55:05.795763839 (535h26m5.666846884s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.802905938 (535h26m5.673988983s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.803078106 (535h26m5.674161151s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.812284336 (535h26m5.683367381s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.829206028 (535h26m5.700289073s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.835000362 (535h26m5.706083397s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.836389036 (535h26m5.707472071s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.843026765 (535h26m5.7141098s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.859561069 (535h26m5.730644104s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.871635229 (535h26m5.742718274s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.080787678 (535h26m5.951870724s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.094740707 (535h26m5.965823752s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122401899 (535h26m5.993484944s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122423880 (535h26m5.993506925s) [seg_pos={0 0}, upload=469598]: Segment reached optimal threshold of 13 pieces. Dec 27 22:55:06.122791581 (535h26m5.993874616s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection Dec 27 22:55:06.122800467 (535h26m5.993883502s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.122806378 (535h26m5.993889413s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.122825335 (535h26m5.99390838s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123315488 (535h26m5.994398523s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.123323623 (535h26m5.994406668s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123341488 (535h26m5.994424533s) [seg_pos={0 0}, upload=469598]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:55:06.123451447 (535h26m5.994534492s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:55:06.123459001 (535h26m5.994542046s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:55:06.413593572 (535h26m6.284676607s) [upload=469598]: Done uploading segments... Dec 27 22:55:06.413613590 (535h26m6.284696626s) [upload=469598]: Upload finished. err: <nil> Duration: 5.922124324s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 3.56487793s trace: https://jaeger.dev.storj.io/trace/4bb85e67f0d4ec1d FAIL (2025-12-27 23:10:15 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.564907286s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..X.X...X.....X...X..X....X.......X.....X...................] duration: 5.974941403s trace: https://jaeger.dev.storj.io/trace/63027bab39a48860 FAIL (2025-12-27 23:09:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA) context deadline exceeded ======LOGS====== Dec 27 22:55:03.533152286 (535h26m3.404235331s) [upload=469598]: Starting upload Dec 27 22:55:03.533478477 (535h26m3.404561522s) [upload=469598]: Done starting upload Dec 27 22:55:03.533881645 (535h26m3.40496469s) [upload=469598]: Uploading segments... Dec 27 22:55:03.544825083 (535h26m3.415908128s) [upload=469598]: Got next segment. Inline: false Dec 27 22:55:03.544990097 (535h26m3.416073142s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:55:03.544999244 (535h26m3.41608229s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:55:03.791379841 (535h26m3.662462886s) [seg_pos={0 0}, upload=469598]: Begin upload segment... Dec 27 22:55:03.791445777 (535h26m3.662528812s) [seg_pos={0 0}, upload=469598]: Done begin upload segment. Dec 27 22:55:03.791485131 (535h26m3.662568176s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.791591594 (535h26m3.662674639s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.793523884 (535h26m3.664606939s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.794185644 (535h26m3.665268689s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.795222378 (535h26m3.666305434s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797183563 (535h26m3.668266618s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797315324 (535h26m3.668398379s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.799339678 (535h26m3.670422733s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.800966926 (535h26m3.672049981s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.802493253 (535h26m3.673576298s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.819392662 (535h26m3.690475708s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.823879014 (535h26m3.694962069s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.828981661 (535h26m3.700064716s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.830805173 (535h26m3.701888238s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.832266125 (535h26m3.70334918s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.836751787 (535h26m3.707834832s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:05.507079228 (535h26m5.378162263s) [upload=469598]: Next returned nil segment Dec 27 22:55:05.507093797 (535h26m5.378176842s) [upload=469598]: Waiting for error group managing segments... Dec 27 22:55:05.795763839 (535h26m5.666846884s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.802905938 (535h26m5.673988983s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.803078106 (535h26m5.674161151s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.812284336 (535h26m5.683367381s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.829206028 (535h26m5.700289073s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.835000362 (535h26m5.706083397s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.836389036 (535h26m5.707472071s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.843026765 (535h26m5.7141098s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.859561069 (535h26m5.730644104s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.871635229 (535h26m5.742718274s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.080787678 (535h26m5.951870724s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.094740707 (535h26m5.965823752s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122401899 (535h26m5.993484944s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122423880 (535h26m5.993506925s) [seg_pos={0 0}, upload=469598]: Segment reached optimal threshold of 13 pieces. Dec 27 22:55:06.122791581 (535h26m5.993874616s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection Dec 27 22:55:06.122800467 (535h26m5.993883502s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.122806378 (535h26m5.993889413s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.122825335 (535h26m5.99390838s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123315488 (535h26m5.994398523s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.123323623 (535h26m5.994406668s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123341488 (535h26m5.994424533s) [seg_pos={0 0}, upload=469598]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:55:06.123451447 (535h26m5.994534492s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:55:06.123459001 (535h26m5.994542046s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:55:06.413593572 (535h26m6.284676607s) [upload=469598]: Done uploading segments... Dec 27 22:55:06.413613590 (535h26m6.284696626s) [upload=469598]: Upload finished. err: <nil> Duration: 5.974971381s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 3.624572858s trace: https://jaeger.dev.storj.io/trace/4bb85e67f0d4ec1d FAIL (2025-12-27 23:09:00 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.624594149s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 3.555419109s trace: https://jaeger.dev.storj.io/trace/4bb85e67f0d4ec1d FAIL (2025-12-27 23:08:35 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.555459134s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 3.559484559s trace: https://jaeger.dev.storj.io/trace/4bb85e67f0d4ec1d FAIL (2025-12-27 23:08:19 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.559512011s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...X.X...X.....X...X..X....X.......X.....X..................] duration: 15.812653245s trace: https://jaeger.dev.storj.io/trace/63027bab39a48860 FAIL (2025-12-27 23:08:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA) context deadline exceeded ======LOGS====== Dec 27 22:55:03.533152286 (535h26m3.404235331s) [upload=469598]: Starting upload Dec 27 22:55:03.533478477 (535h26m3.404561522s) [upload=469598]: Done starting upload Dec 27 22:55:03.533881645 (535h26m3.40496469s) [upload=469598]: Uploading segments... Dec 27 22:55:03.544825083 (535h26m3.415908128s) [upload=469598]: Got next segment. Inline: false Dec 27 22:55:03.544990097 (535h26m3.416073142s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:55:03.544999244 (535h26m3.41608229s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:55:03.791379841 (535h26m3.662462886s) [seg_pos={0 0}, upload=469598]: Begin upload segment... Dec 27 22:55:03.791445777 (535h26m3.662528812s) [seg_pos={0 0}, upload=469598]: Done begin upload segment. Dec 27 22:55:03.791485131 (535h26m3.662568176s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.791591594 (535h26m3.662674639s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.793523884 (535h26m3.664606939s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.794185644 (535h26m3.665268689s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.795222378 (535h26m3.666305434s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797183563 (535h26m3.668266618s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797315324 (535h26m3.668398379s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.799339678 (535h26m3.670422733s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.800966926 (535h26m3.672049981s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.802493253 (535h26m3.673576298s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.819392662 (535h26m3.690475708s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.823879014 (535h26m3.694962069s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.828981661 (535h26m3.700064716s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.830805173 (535h26m3.701888238s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.832266125 (535h26m3.70334918s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.836751787 (535h26m3.707834832s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:05.507079228 (535h26m5.378162263s) [upload=469598]: Next returned nil segment Dec 27 22:55:05.507093797 (535h26m5.378176842s) [upload=469598]: Waiting for error group managing segments... Dec 27 22:55:05.795763839 (535h26m5.666846884s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.802905938 (535h26m5.673988983s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.803078106 (535h26m5.674161151s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.812284336 (535h26m5.683367381s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.829206028 (535h26m5.700289073s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.835000362 (535h26m5.706083397s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.836389036 (535h26m5.707472071s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.843026765 (535h26m5.7141098s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.859561069 (535h26m5.730644104s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.871635229 (535h26m5.742718274s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.080787678 (535h26m5.951870724s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.094740707 (535h26m5.965823752s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122401899 (535h26m5.993484944s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122423880 (535h26m5.993506925s) [seg_pos={0 0}, upload=469598]: Segment reached optimal threshold of 13 pieces. Dec 27 22:55:06.122791581 (535h26m5.993874616s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection Dec 27 22:55:06.122800467 (535h26m5.993883502s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.122806378 (535h26m5.993889413s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.122825335 (535h26m5.99390838s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123315488 (535h26m5.994398523s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.123323623 (535h26m5.994406668s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123341488 (535h26m5.994424533s) [seg_pos={0 0}, upload=469598]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:55:06.123451447 (535h26m5.994534492s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:55:06.123459001 (535h26m5.994542046s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:55:06.413593572 (535h26m6.284676607s) [upload=469598]: Done uploading segments... Dec 27 22:55:06.413613590 (535h26m6.284696626s) [upload=469598]: Upload finished. err: <nil> Duration: 15.812678123s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [....X.X...X.....X...X..X....X.......X.....X.................] duration: 13.69891751s trace: https://jaeger.dev.storj.io/trace/63027bab39a48860 FAIL (2025-12-27 23:07:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA) context deadline exceeded ======LOGS====== Dec 27 22:55:03.533152286 (535h26m3.404235331s) [upload=469598]: Starting upload Dec 27 22:55:03.533478477 (535h26m3.404561522s) [upload=469598]: Done starting upload Dec 27 22:55:03.533881645 (535h26m3.40496469s) [upload=469598]: Uploading segments... Dec 27 22:55:03.544825083 (535h26m3.415908128s) [upload=469598]: Got next segment. Inline: false Dec 27 22:55:03.544990097 (535h26m3.416073142s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:55:03.544999244 (535h26m3.41608229s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:55:03.791379841 (535h26m3.662462886s) [seg_pos={0 0}, upload=469598]: Begin upload segment... Dec 27 22:55:03.791445777 (535h26m3.662528812s) [seg_pos={0 0}, upload=469598]: Done begin upload segment. Dec 27 22:55:03.791485131 (535h26m3.662568176s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.791591594 (535h26m3.662674639s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.793523884 (535h26m3.664606939s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.794185644 (535h26m3.665268689s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.795222378 (535h26m3.666305434s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797183563 (535h26m3.668266618s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797315324 (535h26m3.668398379s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.799339678 (535h26m3.670422733s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.800966926 (535h26m3.672049981s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.802493253 (535h26m3.673576298s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.819392662 (535h26m3.690475708s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.823879014 (535h26m3.694962069s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.828981661 (535h26m3.700064716s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.830805173 (535h26m3.701888238s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.832266125 (535h26m3.70334918s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.836751787 (535h26m3.707834832s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:05.507079228 (535h26m5.378162263s) [upload=469598]: Next returned nil segment Dec 27 22:55:05.507093797 (535h26m5.378176842s) [upload=469598]: Waiting for error group managing segments... Dec 27 22:55:05.795763839 (535h26m5.666846884s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.802905938 (535h26m5.673988983s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.803078106 (535h26m5.674161151s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.812284336 (535h26m5.683367381s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.829206028 (535h26m5.700289073s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.835000362 (535h26m5.706083397s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.836389036 (535h26m5.707472071s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.843026765 (535h26m5.7141098s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.859561069 (535h26m5.730644104s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.871635229 (535h26m5.742718274s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.080787678 (535h26m5.951870724s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.094740707 (535h26m5.965823752s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122401899 (535h26m5.993484944s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122423880 (535h26m5.993506925s) [seg_pos={0 0}, upload=469598]: Segment reached optimal threshold of 13 pieces. Dec 27 22:55:06.122791581 (535h26m5.993874616s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection Dec 27 22:55:06.122800467 (535h26m5.993883502s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.122806378 (535h26m5.993889413s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.122825335 (535h26m5.99390838s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123315488 (535h26m5.994398523s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.123323623 (535h26m5.994406668s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123341488 (535h26m5.994424533s) [seg_pos={0 0}, upload=469598]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:55:06.123451447 (535h26m5.994534492s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:55:06.123459001 (535h26m5.994542046s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:55:06.413593572 (535h26m6.284676607s) [upload=469598]: Done uploading segments... Dec 27 22:55:06.413613590 (535h26m6.284696626s) [upload=469598]: Upload finished. err: <nil> Duration: 13.69894373s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.....X.X...X.....X...X..X....X.......X.....X................] duration: 5.873862804s trace: https://jaeger.dev.storj.io/trace/63027bab39a48860 FAIL (2025-12-27 23:07:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA) context deadline exceeded ======LOGS====== Dec 27 22:55:03.533152286 (535h26m3.404235331s) [upload=469598]: Starting upload Dec 27 22:55:03.533478477 (535h26m3.404561522s) [upload=469598]: Done starting upload Dec 27 22:55:03.533881645 (535h26m3.40496469s) [upload=469598]: Uploading segments... Dec 27 22:55:03.544825083 (535h26m3.415908128s) [upload=469598]: Got next segment. Inline: false Dec 27 22:55:03.544990097 (535h26m3.416073142s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:55:03.544999244 (535h26m3.41608229s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:55:03.791379841 (535h26m3.662462886s) [seg_pos={0 0}, upload=469598]: Begin upload segment... Dec 27 22:55:03.791445777 (535h26m3.662528812s) [seg_pos={0 0}, upload=469598]: Done begin upload segment. Dec 27 22:55:03.791485131 (535h26m3.662568176s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.791591594 (535h26m3.662674639s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.793523884 (535h26m3.664606939s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.794185644 (535h26m3.665268689s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.795222378 (535h26m3.666305434s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797183563 (535h26m3.668266618s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797315324 (535h26m3.668398379s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.799339678 (535h26m3.670422733s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.800966926 (535h26m3.672049981s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.802493253 (535h26m3.673576298s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.819392662 (535h26m3.690475708s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.823879014 (535h26m3.694962069s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.828981661 (535h26m3.700064716s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.830805173 (535h26m3.701888238s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.832266125 (535h26m3.70334918s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.836751787 (535h26m3.707834832s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:05.507079228 (535h26m5.378162263s) [upload=469598]: Next returned nil segment Dec 27 22:55:05.507093797 (535h26m5.378176842s) [upload=469598]: Waiting for error group managing segments... Dec 27 22:55:05.795763839 (535h26m5.666846884s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.802905938 (535h26m5.673988983s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.803078106 (535h26m5.674161151s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.812284336 (535h26m5.683367381s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.829206028 (535h26m5.700289073s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.835000362 (535h26m5.706083397s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.836389036 (535h26m5.707472071s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.843026765 (535h26m5.7141098s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.859561069 (535h26m5.730644104s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.871635229 (535h26m5.742718274s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.080787678 (535h26m5.951870724s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.094740707 (535h26m5.965823752s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122401899 (535h26m5.993484944s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122423880 (535h26m5.993506925s) [seg_pos={0 0}, upload=469598]: Segment reached optimal threshold of 13 pieces. Dec 27 22:55:06.122791581 (535h26m5.993874616s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection Dec 27 22:55:06.122800467 (535h26m5.993883502s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.122806378 (535h26m5.993889413s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.122825335 (535h26m5.99390838s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123315488 (535h26m5.994398523s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.123323623 (535h26m5.994406668s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123341488 (535h26m5.994424533s) [seg_pos={0 0}, upload=469598]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:55:06.123451447 (535h26m5.994534492s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:55:06.123459001 (535h26m5.994542046s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:55:06.413593572 (535h26m6.284676607s) [upload=469598]: Done uploading segments... Dec 27 22:55:06.413613590 (535h26m6.284696626s) [upload=469598]: Upload finished. err: <nil> Duration: 5.873886269s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 3.544410997s trace: https://jaeger.dev.storj.io/trace/4bb85e67f0d4ec1d FAIL (2025-12-27 23:07:06 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.54445438s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 3.379572187s trace: https://jaeger.dev.storj.io/trace/4bb85e67f0d4ec1d FAIL (2025-12-27 23:06:46 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.379594889s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [......X.X...X.....X...X..X....X.......X.....X...............] duration: 6.232791867s trace: https://jaeger.dev.storj.io/trace/63027bab39a48860 FAIL (2025-12-27 23:06:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA) context deadline exceeded ======LOGS====== Dec 27 22:55:03.533152286 (535h26m3.404235331s) [upload=469598]: Starting upload Dec 27 22:55:03.533478477 (535h26m3.404561522s) [upload=469598]: Done starting upload Dec 27 22:55:03.533881645 (535h26m3.40496469s) [upload=469598]: Uploading segments... Dec 27 22:55:03.544825083 (535h26m3.415908128s) [upload=469598]: Got next segment. Inline: false Dec 27 22:55:03.544990097 (535h26m3.416073142s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:55:03.544999244 (535h26m3.41608229s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:55:03.791379841 (535h26m3.662462886s) [seg_pos={0 0}, upload=469598]: Begin upload segment... Dec 27 22:55:03.791445777 (535h26m3.662528812s) [seg_pos={0 0}, upload=469598]: Done begin upload segment. Dec 27 22:55:03.791485131 (535h26m3.662568176s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.791591594 (535h26m3.662674639s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.793523884 (535h26m3.664606939s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.794185644 (535h26m3.665268689s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.795222378 (535h26m3.666305434s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797183563 (535h26m3.668266618s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797315324 (535h26m3.668398379s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.799339678 (535h26m3.670422733s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.800966926 (535h26m3.672049981s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.802493253 (535h26m3.673576298s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.819392662 (535h26m3.690475708s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.823879014 (535h26m3.694962069s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.828981661 (535h26m3.700064716s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.830805173 (535h26m3.701888238s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.832266125 (535h26m3.70334918s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.836751787 (535h26m3.707834832s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:05.507079228 (535h26m5.378162263s) [upload=469598]: Next returned nil segment Dec 27 22:55:05.507093797 (535h26m5.378176842s) [upload=469598]: Waiting for error group managing segments... Dec 27 22:55:05.795763839 (535h26m5.666846884s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.802905938 (535h26m5.673988983s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.803078106 (535h26m5.674161151s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.812284336 (535h26m5.683367381s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.829206028 (535h26m5.700289073s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.835000362 (535h26m5.706083397s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.836389036 (535h26m5.707472071s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.843026765 (535h26m5.7141098s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.859561069 (535h26m5.730644104s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.871635229 (535h26m5.742718274s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.080787678 (535h26m5.951870724s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.094740707 (535h26m5.965823752s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122401899 (535h26m5.993484944s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122423880 (535h26m5.993506925s) [seg_pos={0 0}, upload=469598]: Segment reached optimal threshold of 13 pieces. Dec 27 22:55:06.122791581 (535h26m5.993874616s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection Dec 27 22:55:06.122800467 (535h26m5.993883502s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.122806378 (535h26m5.993889413s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.122825335 (535h26m5.99390838s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123315488 (535h26m5.994398523s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.123323623 (535h26m5.994406668s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123341488 (535h26m5.994424533s) [seg_pos={0 0}, upload=469598]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:55:06.123451447 (535h26m5.994534492s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:55:06.123459001 (535h26m5.994542046s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:55:06.413593572 (535h26m6.284676607s) [upload=469598]: Done uploading segments... Dec 27 22:55:06.413613590 (535h26m6.284696626s) [upload=469598]: Upload finished. err: <nil> Duration: 6.232819258s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 3.587214121s trace: https://jaeger.dev.storj.io/trace/4bb85e67f0d4ec1d FAIL (2025-12-27 23:06:28 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.587240913s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.......X.X...X.....X...X..X....X.......X.....X..............] duration: 6.692923764s trace: https://jaeger.dev.storj.io/trace/63027bab39a48860 FAIL (2025-12-27 23:06:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA) context deadline exceeded ======LOGS====== Dec 27 22:55:03.533152286 (535h26m3.404235331s) [upload=469598]: Starting upload Dec 27 22:55:03.533478477 (535h26m3.404561522s) [upload=469598]: Done starting upload Dec 27 22:55:03.533881645 (535h26m3.40496469s) [upload=469598]: Uploading segments... Dec 27 22:55:03.544825083 (535h26m3.415908128s) [upload=469598]: Got next segment. Inline: false Dec 27 22:55:03.544990097 (535h26m3.416073142s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:55:03.544999244 (535h26m3.41608229s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:55:03.791379841 (535h26m3.662462886s) [seg_pos={0 0}, upload=469598]: Begin upload segment... Dec 27 22:55:03.791445777 (535h26m3.662528812s) [seg_pos={0 0}, upload=469598]: Done begin upload segment. Dec 27 22:55:03.791485131 (535h26m3.662568176s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.791591594 (535h26m3.662674639s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.793523884 (535h26m3.664606939s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.794185644 (535h26m3.665268689s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.795222378 (535h26m3.666305434s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797183563 (535h26m3.668266618s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797315324 (535h26m3.668398379s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.799339678 (535h26m3.670422733s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.800966926 (535h26m3.672049981s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.802493253 (535h26m3.673576298s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.819392662 (535h26m3.690475708s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.823879014 (535h26m3.694962069s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.828981661 (535h26m3.700064716s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.830805173 (535h26m3.701888238s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.832266125 (535h26m3.70334918s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.836751787 (535h26m3.707834832s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:05.507079228 (535h26m5.378162263s) [upload=469598]: Next returned nil segment Dec 27 22:55:05.507093797 (535h26m5.378176842s) [upload=469598]: Waiting for error group managing segments... Dec 27 22:55:05.795763839 (535h26m5.666846884s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.802905938 (535h26m5.673988983s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.803078106 (535h26m5.674161151s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.812284336 (535h26m5.683367381s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.829206028 (535h26m5.700289073s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.835000362 (535h26m5.706083397s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.836389036 (535h26m5.707472071s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.843026765 (535h26m5.7141098s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.859561069 (535h26m5.730644104s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.871635229 (535h26m5.742718274s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.080787678 (535h26m5.951870724s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.094740707 (535h26m5.965823752s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122401899 (535h26m5.993484944s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122423880 (535h26m5.993506925s) [seg_pos={0 0}, upload=469598]: Segment reached optimal threshold of 13 pieces. Dec 27 22:55:06.122791581 (535h26m5.993874616s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection Dec 27 22:55:06.122800467 (535h26m5.993883502s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.122806378 (535h26m5.993889413s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.122825335 (535h26m5.99390838s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123315488 (535h26m5.994398523s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.123323623 (535h26m5.994406668s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123341488 (535h26m5.994424533s) [seg_pos={0 0}, upload=469598]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:55:06.123451447 (535h26m5.994534492s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:55:06.123459001 (535h26m5.994542046s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:55:06.413593572 (535h26m6.284676607s) [upload=469598]: Done uploading segments... Dec 27 22:55:06.413613590 (535h26m6.284696626s) [upload=469598]: Upload finished. err: <nil> Duration: 6.692950956s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [........X.X...X.....X...X..X....X.......X.....X.............] duration: 8.119763887s trace: https://jaeger.dev.storj.io/trace/63027bab39a48860 FAIL (2025-12-27 23:05:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA) context deadline exceeded ======LOGS====== Dec 27 22:55:03.533152286 (535h26m3.404235331s) [upload=469598]: Starting upload Dec 27 22:55:03.533478477 (535h26m3.404561522s) [upload=469598]: Done starting upload Dec 27 22:55:03.533881645 (535h26m3.40496469s) [upload=469598]: Uploading segments... Dec 27 22:55:03.544825083 (535h26m3.415908128s) [upload=469598]: Got next segment. Inline: false Dec 27 22:55:03.544990097 (535h26m3.416073142s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:55:03.544999244 (535h26m3.41608229s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:55:03.791379841 (535h26m3.662462886s) [seg_pos={0 0}, upload=469598]: Begin upload segment... Dec 27 22:55:03.791445777 (535h26m3.662528812s) [seg_pos={0 0}, upload=469598]: Done begin upload segment. Dec 27 22:55:03.791485131 (535h26m3.662568176s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.791591594 (535h26m3.662674639s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.793523884 (535h26m3.664606939s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.794185644 (535h26m3.665268689s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.795222378 (535h26m3.666305434s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797183563 (535h26m3.668266618s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797315324 (535h26m3.668398379s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.799339678 (535h26m3.670422733s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.800966926 (535h26m3.672049981s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.802493253 (535h26m3.673576298s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.819392662 (535h26m3.690475708s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.823879014 (535h26m3.694962069s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.828981661 (535h26m3.700064716s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.830805173 (535h26m3.701888238s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.832266125 (535h26m3.70334918s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.836751787 (535h26m3.707834832s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:05.507079228 (535h26m5.378162263s) [upload=469598]: Next returned nil segment Dec 27 22:55:05.507093797 (535h26m5.378176842s) [upload=469598]: Waiting for error group managing segments... Dec 27 22:55:05.795763839 (535h26m5.666846884s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.802905938 (535h26m5.673988983s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.803078106 (535h26m5.674161151s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.812284336 (535h26m5.683367381s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.829206028 (535h26m5.700289073s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.835000362 (535h26m5.706083397s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.836389036 (535h26m5.707472071s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.843026765 (535h26m5.7141098s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.859561069 (535h26m5.730644104s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.871635229 (535h26m5.742718274s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.080787678 (535h26m5.951870724s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.094740707 (535h26m5.965823752s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122401899 (535h26m5.993484944s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122423880 (535h26m5.993506925s) [seg_pos={0 0}, upload=469598]: Segment reached optimal threshold of 13 pieces. Dec 27 22:55:06.122791581 (535h26m5.993874616s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection Dec 27 22:55:06.122800467 (535h26m5.993883502s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.122806378 (535h26m5.993889413s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.122825335 (535h26m5.99390838s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123315488 (535h26m5.994398523s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.123323623 (535h26m5.994406668s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123341488 (535h26m5.994424533s) [seg_pos={0 0}, upload=469598]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:55:06.123451447 (535h26m5.994534492s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:55:06.123459001 (535h26m5.994542046s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:55:06.413593572 (535h26m6.284676607s) [upload=469598]: Done uploading segments... Dec 27 22:55:06.413613590 (535h26m6.284696626s) [upload=469598]: Upload finished. err: <nil> Duration: 8.119792212s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 3.504833565s trace: https://jaeger.dev.storj.io/trace/4bb85e67f0d4ec1d FAIL (2025-12-27 23:05:41 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.504931663s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.........X.X...X.....X...X..X....X.......X.....X............] duration: 5.831165383s trace: https://jaeger.dev.storj.io/trace/63027bab39a48860 FAIL (2025-12-27 23:04:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA) context deadline exceeded ======LOGS====== Dec 27 22:55:03.533152286 (535h26m3.404235331s) [upload=469598]: Starting upload Dec 27 22:55:03.533478477 (535h26m3.404561522s) [upload=469598]: Done starting upload Dec 27 22:55:03.533881645 (535h26m3.40496469s) [upload=469598]: Uploading segments... Dec 27 22:55:03.544825083 (535h26m3.415908128s) [upload=469598]: Got next segment. Inline: false Dec 27 22:55:03.544990097 (535h26m3.416073142s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:55:03.544999244 (535h26m3.41608229s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:55:03.791379841 (535h26m3.662462886s) [seg_pos={0 0}, upload=469598]: Begin upload segment... Dec 27 22:55:03.791445777 (535h26m3.662528812s) [seg_pos={0 0}, upload=469598]: Done begin upload segment. Dec 27 22:55:03.791485131 (535h26m3.662568176s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.791591594 (535h26m3.662674639s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.793523884 (535h26m3.664606939s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.794185644 (535h26m3.665268689s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.795222378 (535h26m3.666305434s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797183563 (535h26m3.668266618s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797315324 (535h26m3.668398379s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.799339678 (535h26m3.670422733s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.800966926 (535h26m3.672049981s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.802493253 (535h26m3.673576298s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.819392662 (535h26m3.690475708s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.823879014 (535h26m3.694962069s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.828981661 (535h26m3.700064716s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.830805173 (535h26m3.701888238s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.832266125 (535h26m3.70334918s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.836751787 (535h26m3.707834832s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:05.507079228 (535h26m5.378162263s) [upload=469598]: Next returned nil segment Dec 27 22:55:05.507093797 (535h26m5.378176842s) [upload=469598]: Waiting for error group managing segments... Dec 27 22:55:05.795763839 (535h26m5.666846884s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.802905938 (535h26m5.673988983s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.803078106 (535h26m5.674161151s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.812284336 (535h26m5.683367381s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.829206028 (535h26m5.700289073s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.835000362 (535h26m5.706083397s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.836389036 (535h26m5.707472071s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.843026765 (535h26m5.7141098s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.859561069 (535h26m5.730644104s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.871635229 (535h26m5.742718274s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.080787678 (535h26m5.951870724s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.094740707 (535h26m5.965823752s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122401899 (535h26m5.993484944s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122423880 (535h26m5.993506925s) [seg_pos={0 0}, upload=469598]: Segment reached optimal threshold of 13 pieces. Dec 27 22:55:06.122791581 (535h26m5.993874616s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection Dec 27 22:55:06.122800467 (535h26m5.993883502s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.122806378 (535h26m5.993889413s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.122825335 (535h26m5.99390838s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123315488 (535h26m5.994398523s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.123323623 (535h26m5.994406668s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123341488 (535h26m5.994424533s) [seg_pos={0 0}, upload=469598]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:55:06.123451447 (535h26m5.994534492s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:55:06.123459001 (535h26m5.994542046s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:55:06.413593572 (535h26m6.284676607s) [upload=469598]: Done uploading segments... Dec 27 22:55:06.413613590 (535h26m6.284696626s) [upload=469598]: Upload finished. err: <nil> Duration: 5.831190981s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 3.521142562s trace: https://jaeger.dev.storj.io/trace/4bb85e67f0d4ec1d FAIL (2025-12-27 23:04:30 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.521165736s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 3.527492991s trace: https://jaeger.dev.storj.io/trace/4bb85e67f0d4ec1d FAIL (2025-12-27 23:04:10 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.527518149s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..........X.X...X.....X...X..X....X.......X.....X...........] duration: 5.963016433s trace: https://jaeger.dev.storj.io/trace/63027bab39a48860 FAIL (2025-12-27 23:03:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA) context deadline exceeded ======LOGS====== Dec 27 22:55:03.533152286 (535h26m3.404235331s) [upload=469598]: Starting upload Dec 27 22:55:03.533478477 (535h26m3.404561522s) [upload=469598]: Done starting upload Dec 27 22:55:03.533881645 (535h26m3.40496469s) [upload=469598]: Uploading segments... Dec 27 22:55:03.544825083 (535h26m3.415908128s) [upload=469598]: Got next segment. Inline: false Dec 27 22:55:03.544990097 (535h26m3.416073142s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:55:03.544999244 (535h26m3.41608229s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:55:03.791379841 (535h26m3.662462886s) [seg_pos={0 0}, upload=469598]: Begin upload segment... Dec 27 22:55:03.791445777 (535h26m3.662528812s) [seg_pos={0 0}, upload=469598]: Done begin upload segment. Dec 27 22:55:03.791485131 (535h26m3.662568176s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.791591594 (535h26m3.662674639s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.793523884 (535h26m3.664606939s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.794185644 (535h26m3.665268689s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.795222378 (535h26m3.666305434s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797183563 (535h26m3.668266618s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797315324 (535h26m3.668398379s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.799339678 (535h26m3.670422733s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.800966926 (535h26m3.672049981s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.802493253 (535h26m3.673576298s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.819392662 (535h26m3.690475708s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.823879014 (535h26m3.694962069s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.828981661 (535h26m3.700064716s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.830805173 (535h26m3.701888238s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.832266125 (535h26m3.70334918s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.836751787 (535h26m3.707834832s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:05.507079228 (535h26m5.378162263s) [upload=469598]: Next returned nil segment Dec 27 22:55:05.507093797 (535h26m5.378176842s) [upload=469598]: Waiting for error group managing segments... Dec 27 22:55:05.795763839 (535h26m5.666846884s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.802905938 (535h26m5.673988983s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.803078106 (535h26m5.674161151s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.812284336 (535h26m5.683367381s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.829206028 (535h26m5.700289073s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.835000362 (535h26m5.706083397s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.836389036 (535h26m5.707472071s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.843026765 (535h26m5.7141098s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.859561069 (535h26m5.730644104s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.871635229 (535h26m5.742718274s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.080787678 (535h26m5.951870724s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.094740707 (535h26m5.965823752s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122401899 (535h26m5.993484944s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122423880 (535h26m5.993506925s) [seg_pos={0 0}, upload=469598]: Segment reached optimal threshold of 13 pieces. Dec 27 22:55:06.122791581 (535h26m5.993874616s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection Dec 27 22:55:06.122800467 (535h26m5.993883502s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.122806378 (535h26m5.993889413s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.122825335 (535h26m5.99390838s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123315488 (535h26m5.994398523s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.123323623 (535h26m5.994406668s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123341488 (535h26m5.994424533s) [seg_pos={0 0}, upload=469598]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:55:06.123451447 (535h26m5.994534492s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:55:06.123459001 (535h26m5.994542046s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:55:06.413593572 (535h26m6.284676607s) [upload=469598]: Done uploading segments... Dec 27 22:55:06.413613590 (535h26m6.284696626s) [upload=469598]: Upload finished. err: <nil> Duration: 5.963039708s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.000996744s trace: https://jaeger.dev.storj.io/trace/4bb85e67f0d4ec1d FAIL (2025-12-27 23:03:14 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001023435s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...........X.X...X.....X...X..X....X.......X.....X..........] duration: 5.931804876s trace: https://jaeger.dev.storj.io/trace/63027bab39a48860 FAIL (2025-12-27 23:02:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA) context deadline exceeded ======LOGS====== Dec 27 22:55:03.533152286 (535h26m3.404235331s) [upload=469598]: Starting upload Dec 27 22:55:03.533478477 (535h26m3.404561522s) [upload=469598]: Done starting upload Dec 27 22:55:03.533881645 (535h26m3.40496469s) [upload=469598]: Uploading segments... Dec 27 22:55:03.544825083 (535h26m3.415908128s) [upload=469598]: Got next segment. Inline: false Dec 27 22:55:03.544990097 (535h26m3.416073142s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:55:03.544999244 (535h26m3.41608229s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:55:03.791379841 (535h26m3.662462886s) [seg_pos={0 0}, upload=469598]: Begin upload segment... Dec 27 22:55:03.791445777 (535h26m3.662528812s) [seg_pos={0 0}, upload=469598]: Done begin upload segment. Dec 27 22:55:03.791485131 (535h26m3.662568176s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.791591594 (535h26m3.662674639s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.793523884 (535h26m3.664606939s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.794185644 (535h26m3.665268689s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.795222378 (535h26m3.666305434s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797183563 (535h26m3.668266618s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797315324 (535h26m3.668398379s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.799339678 (535h26m3.670422733s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.800966926 (535h26m3.672049981s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.802493253 (535h26m3.673576298s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.819392662 (535h26m3.690475708s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.823879014 (535h26m3.694962069s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.828981661 (535h26m3.700064716s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.830805173 (535h26m3.701888238s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.832266125 (535h26m3.70334918s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.836751787 (535h26m3.707834832s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:05.507079228 (535h26m5.378162263s) [upload=469598]: Next returned nil segment Dec 27 22:55:05.507093797 (535h26m5.378176842s) [upload=469598]: Waiting for error group managing segments... Dec 27 22:55:05.795763839 (535h26m5.666846884s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.802905938 (535h26m5.673988983s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.803078106 (535h26m5.674161151s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.812284336 (535h26m5.683367381s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.829206028 (535h26m5.700289073s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.835000362 (535h26m5.706083397s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.836389036 (535h26m5.707472071s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.843026765 (535h26m5.7141098s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.859561069 (535h26m5.730644104s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.871635229 (535h26m5.742718274s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.080787678 (535h26m5.951870724s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.094740707 (535h26m5.965823752s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122401899 (535h26m5.993484944s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122423880 (535h26m5.993506925s) [seg_pos={0 0}, upload=469598]: Segment reached optimal threshold of 13 pieces. Dec 27 22:55:06.122791581 (535h26m5.993874616s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection Dec 27 22:55:06.122800467 (535h26m5.993883502s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.122806378 (535h26m5.993889413s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.122825335 (535h26m5.99390838s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123315488 (535h26m5.994398523s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.123323623 (535h26m5.994406668s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123341488 (535h26m5.994424533s) [seg_pos={0 0}, upload=469598]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:55:06.123451447 (535h26m5.994534492s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:55:06.123459001 (535h26m5.994542046s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:55:06.413593572 (535h26m6.284676607s) [upload=469598]: Done uploading segments... Dec 27 22:55:06.413613590 (535h26m6.284696626s) [upload=469598]: Upload finished. err: <nil> Duration: 5.931828632s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [............X.X...X.....X...X..X....X.......X.....X.........] duration: 6.167364721s trace: https://jaeger.dev.storj.io/trace/63027bab39a48860 FAIL (2025-12-27 23:02:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA) context deadline exceeded ======LOGS====== Dec 27 22:55:03.533152286 (535h26m3.404235331s) [upload=469598]: Starting upload Dec 27 22:55:03.533478477 (535h26m3.404561522s) [upload=469598]: Done starting upload Dec 27 22:55:03.533881645 (535h26m3.40496469s) [upload=469598]: Uploading segments... Dec 27 22:55:03.544825083 (535h26m3.415908128s) [upload=469598]: Got next segment. Inline: false Dec 27 22:55:03.544990097 (535h26m3.416073142s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:55:03.544999244 (535h26m3.41608229s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:55:03.791379841 (535h26m3.662462886s) [seg_pos={0 0}, upload=469598]: Begin upload segment... Dec 27 22:55:03.791445777 (535h26m3.662528812s) [seg_pos={0 0}, upload=469598]: Done begin upload segment. Dec 27 22:55:03.791485131 (535h26m3.662568176s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.791591594 (535h26m3.662674639s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.793523884 (535h26m3.664606939s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.794185644 (535h26m3.665268689s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.795222378 (535h26m3.666305434s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797183563 (535h26m3.668266618s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797315324 (535h26m3.668398379s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.799339678 (535h26m3.670422733s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.800966926 (535h26m3.672049981s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.802493253 (535h26m3.673576298s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.819392662 (535h26m3.690475708s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.823879014 (535h26m3.694962069s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.828981661 (535h26m3.700064716s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.830805173 (535h26m3.701888238s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.832266125 (535h26m3.70334918s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.836751787 (535h26m3.707834832s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:05.507079228 (535h26m5.378162263s) [upload=469598]: Next returned nil segment Dec 27 22:55:05.507093797 (535h26m5.378176842s) [upload=469598]: Waiting for error group managing segments... Dec 27 22:55:05.795763839 (535h26m5.666846884s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.802905938 (535h26m5.673988983s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.803078106 (535h26m5.674161151s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.812284336 (535h26m5.683367381s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.829206028 (535h26m5.700289073s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.835000362 (535h26m5.706083397s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.836389036 (535h26m5.707472071s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.843026765 (535h26m5.7141098s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.859561069 (535h26m5.730644104s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.871635229 (535h26m5.742718274s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.080787678 (535h26m5.951870724s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.094740707 (535h26m5.965823752s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122401899 (535h26m5.993484944s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122423880 (535h26m5.993506925s) [seg_pos={0 0}, upload=469598]: Segment reached optimal threshold of 13 pieces. Dec 27 22:55:06.122791581 (535h26m5.993874616s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection Dec 27 22:55:06.122800467 (535h26m5.993883502s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.122806378 (535h26m5.993889413s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.122825335 (535h26m5.99390838s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123315488 (535h26m5.994398523s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.123323623 (535h26m5.994406668s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123341488 (535h26m5.994424533s) [seg_pos={0 0}, upload=469598]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:55:06.123451447 (535h26m5.994534492s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:55:06.123459001 (535h26m5.994542046s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:55:06.413593572 (535h26m6.284676607s) [upload=469598]: Done uploading segments... Dec 27 22:55:06.413613590 (535h26m6.284696626s) [upload=469598]: Upload finished. err: <nil> Duration: 6.167387144s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X............X.X...X.....X...X..X....X.......X.....X........] duration: 5.985993004s trace: https://jaeger.dev.storj.io/trace/63027bab39a48860 FAIL (2025-12-27 23:01:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA) context deadline exceeded ======LOGS====== Dec 27 22:55:03.533152286 (535h26m3.404235331s) [upload=469598]: Starting upload Dec 27 22:55:03.533478477 (535h26m3.404561522s) [upload=469598]: Done starting upload Dec 27 22:55:03.533881645 (535h26m3.40496469s) [upload=469598]: Uploading segments... Dec 27 22:55:03.544825083 (535h26m3.415908128s) [upload=469598]: Got next segment. Inline: false Dec 27 22:55:03.544990097 (535h26m3.416073142s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:55:03.544999244 (535h26m3.41608229s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:55:03.791379841 (535h26m3.662462886s) [seg_pos={0 0}, upload=469598]: Begin upload segment... Dec 27 22:55:03.791445777 (535h26m3.662528812s) [seg_pos={0 0}, upload=469598]: Done begin upload segment. Dec 27 22:55:03.791485131 (535h26m3.662568176s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.791591594 (535h26m3.662674639s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.793523884 (535h26m3.664606939s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.794185644 (535h26m3.665268689s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.795222378 (535h26m3.666305434s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797183563 (535h26m3.668266618s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797315324 (535h26m3.668398379s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.799339678 (535h26m3.670422733s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.800966926 (535h26m3.672049981s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.802493253 (535h26m3.673576298s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.819392662 (535h26m3.690475708s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.823879014 (535h26m3.694962069s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.828981661 (535h26m3.700064716s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.830805173 (535h26m3.701888238s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.832266125 (535h26m3.70334918s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.836751787 (535h26m3.707834832s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:05.507079228 (535h26m5.378162263s) [upload=469598]: Next returned nil segment Dec 27 22:55:05.507093797 (535h26m5.378176842s) [upload=469598]: Waiting for error group managing segments... Dec 27 22:55:05.795763839 (535h26m5.666846884s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.802905938 (535h26m5.673988983s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.803078106 (535h26m5.674161151s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.812284336 (535h26m5.683367381s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.829206028 (535h26m5.700289073s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.835000362 (535h26m5.706083397s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.836389036 (535h26m5.707472071s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.843026765 (535h26m5.7141098s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.859561069 (535h26m5.730644104s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.871635229 (535h26m5.742718274s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.080787678 (535h26m5.951870724s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.094740707 (535h26m5.965823752s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122401899 (535h26m5.993484944s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122423880 (535h26m5.993506925s) [seg_pos={0 0}, upload=469598]: Segment reached optimal threshold of 13 pieces. Dec 27 22:55:06.122791581 (535h26m5.993874616s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection Dec 27 22:55:06.122800467 (535h26m5.993883502s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.122806378 (535h26m5.993889413s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.122825335 (535h26m5.99390838s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123315488 (535h26m5.994398523s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.123323623 (535h26m5.994406668s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123341488 (535h26m5.994424533s) [seg_pos={0 0}, upload=469598]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:55:06.123451447 (535h26m5.994534492s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:55:06.123459001 (535h26m5.994542046s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:55:06.413593572 (535h26m6.284676607s) [upload=469598]: Done uploading segments... Dec 27 22:55:06.413613590 (535h26m6.284696626s) [upload=469598]: Upload finished. err: <nil> Duration: 5.986020988s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X............X.X...X.....X...X..X....X.......X.....X.......] duration: 6.171756933s trace: https://jaeger.dev.storj.io/trace/63027bab39a48860 FAIL (2025-12-27 23:00:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA) context deadline exceeded ======LOGS====== Dec 27 22:55:03.533152286 (535h26m3.404235331s) [upload=469598]: Starting upload Dec 27 22:55:03.533478477 (535h26m3.404561522s) [upload=469598]: Done starting upload Dec 27 22:55:03.533881645 (535h26m3.40496469s) [upload=469598]: Uploading segments... Dec 27 22:55:03.544825083 (535h26m3.415908128s) [upload=469598]: Got next segment. Inline: false Dec 27 22:55:03.544990097 (535h26m3.416073142s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:55:03.544999244 (535h26m3.41608229s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:55:03.791379841 (535h26m3.662462886s) [seg_pos={0 0}, upload=469598]: Begin upload segment... Dec 27 22:55:03.791445777 (535h26m3.662528812s) [seg_pos={0 0}, upload=469598]: Done begin upload segment. Dec 27 22:55:03.791485131 (535h26m3.662568176s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.791591594 (535h26m3.662674639s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.793523884 (535h26m3.664606939s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.794185644 (535h26m3.665268689s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.795222378 (535h26m3.666305434s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797183563 (535h26m3.668266618s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797315324 (535h26m3.668398379s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.799339678 (535h26m3.670422733s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.800966926 (535h26m3.672049981s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.802493253 (535h26m3.673576298s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.819392662 (535h26m3.690475708s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.823879014 (535h26m3.694962069s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.828981661 (535h26m3.700064716s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.830805173 (535h26m3.701888238s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.832266125 (535h26m3.70334918s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.836751787 (535h26m3.707834832s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:05.507079228 (535h26m5.378162263s) [upload=469598]: Next returned nil segment Dec 27 22:55:05.507093797 (535h26m5.378176842s) [upload=469598]: Waiting for error group managing segments... Dec 27 22:55:05.795763839 (535h26m5.666846884s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.802905938 (535h26m5.673988983s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.803078106 (535h26m5.674161151s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.812284336 (535h26m5.683367381s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.829206028 (535h26m5.700289073s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.835000362 (535h26m5.706083397s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.836389036 (535h26m5.707472071s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.843026765 (535h26m5.7141098s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.859561069 (535h26m5.730644104s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.871635229 (535h26m5.742718274s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.080787678 (535h26m5.951870724s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.094740707 (535h26m5.965823752s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122401899 (535h26m5.993484944s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122423880 (535h26m5.993506925s) [seg_pos={0 0}, upload=469598]: Segment reached optimal threshold of 13 pieces. Dec 27 22:55:06.122791581 (535h26m5.993874616s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection Dec 27 22:55:06.122800467 (535h26m5.993883502s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.122806378 (535h26m5.993889413s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.122825335 (535h26m5.99390838s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123315488 (535h26m5.994398523s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.123323623 (535h26m5.994406668s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123341488 (535h26m5.994424533s) [seg_pos={0 0}, upload=469598]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:55:06.123451447 (535h26m5.994534492s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:55:06.123459001 (535h26m5.994542046s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:55:06.413593572 (535h26m6.284676607s) [upload=469598]: Done uploading segments... Dec 27 22:55:06.413613590 (535h26m6.284696626s) [upload=469598]: Upload finished. err: <nil> Duration: 6.17186528s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..X............X.X...X.....X...X..X....X.......X.....X......] duration: 18.759918858s trace: https://jaeger.dev.storj.io/trace/63027bab39a48860 FAIL (2025-12-27 23:00:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA) context deadline exceeded ======LOGS====== Dec 27 22:55:03.533152286 (535h26m3.404235331s) [upload=469598]: Starting upload Dec 27 22:55:03.533478477 (535h26m3.404561522s) [upload=469598]: Done starting upload Dec 27 22:55:03.533881645 (535h26m3.40496469s) [upload=469598]: Uploading segments... Dec 27 22:55:03.544825083 (535h26m3.415908128s) [upload=469598]: Got next segment. Inline: false Dec 27 22:55:03.544990097 (535h26m3.416073142s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:55:03.544999244 (535h26m3.41608229s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:55:03.791379841 (535h26m3.662462886s) [seg_pos={0 0}, upload=469598]: Begin upload segment... Dec 27 22:55:03.791445777 (535h26m3.662528812s) [seg_pos={0 0}, upload=469598]: Done begin upload segment. Dec 27 22:55:03.791485131 (535h26m3.662568176s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.791591594 (535h26m3.662674639s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.793523884 (535h26m3.664606939s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.794185644 (535h26m3.665268689s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.795222378 (535h26m3.666305434s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797183563 (535h26m3.668266618s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797315324 (535h26m3.668398379s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.799339678 (535h26m3.670422733s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.800966926 (535h26m3.672049981s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.802493253 (535h26m3.673576298s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.819392662 (535h26m3.690475708s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.823879014 (535h26m3.694962069s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.828981661 (535h26m3.700064716s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.830805173 (535h26m3.701888238s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.832266125 (535h26m3.70334918s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.836751787 (535h26m3.707834832s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:05.507079228 (535h26m5.378162263s) [upload=469598]: Next returned nil segment Dec 27 22:55:05.507093797 (535h26m5.378176842s) [upload=469598]: Waiting for error group managing segments... Dec 27 22:55:05.795763839 (535h26m5.666846884s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.802905938 (535h26m5.673988983s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.803078106 (535h26m5.674161151s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.812284336 (535h26m5.683367381s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.829206028 (535h26m5.700289073s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.835000362 (535h26m5.706083397s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.836389036 (535h26m5.707472071s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.843026765 (535h26m5.7141098s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.859561069 (535h26m5.730644104s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.871635229 (535h26m5.742718274s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.080787678 (535h26m5.951870724s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.094740707 (535h26m5.965823752s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122401899 (535h26m5.993484944s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122423880 (535h26m5.993506925s) [seg_pos={0 0}, upload=469598]: Segment reached optimal threshold of 13 pieces. Dec 27 22:55:06.122791581 (535h26m5.993874616s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection Dec 27 22:55:06.122800467 (535h26m5.993883502s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.122806378 (535h26m5.993889413s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.122825335 (535h26m5.99390838s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123315488 (535h26m5.994398523s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.123323623 (535h26m5.994406668s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123341488 (535h26m5.994424533s) [seg_pos={0 0}, upload=469598]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:55:06.123451447 (535h26m5.994534492s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:55:06.123459001 (535h26m5.994542046s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:55:06.413593572 (535h26m6.284676607s) [upload=469598]: Done uploading segments... Dec 27 22:55:06.413613590 (535h26m6.284696626s) [upload=469598]: Upload finished. err: <nil> Duration: 18.759944908s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...X............X.X...X.....X...X..X....X.......X.....X.....] duration: 6.076367393s trace: https://jaeger.dev.storj.io/trace/63027bab39a48860 FAIL (2025-12-27 22:59:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA) context deadline exceeded ======LOGS====== Dec 27 22:55:03.533152286 (535h26m3.404235331s) [upload=469598]: Starting upload Dec 27 22:55:03.533478477 (535h26m3.404561522s) [upload=469598]: Done starting upload Dec 27 22:55:03.533881645 (535h26m3.40496469s) [upload=469598]: Uploading segments... Dec 27 22:55:03.544825083 (535h26m3.415908128s) [upload=469598]: Got next segment. Inline: false Dec 27 22:55:03.544990097 (535h26m3.416073142s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:55:03.544999244 (535h26m3.41608229s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:55:03.791379841 (535h26m3.662462886s) [seg_pos={0 0}, upload=469598]: Begin upload segment... Dec 27 22:55:03.791445777 (535h26m3.662528812s) [seg_pos={0 0}, upload=469598]: Done begin upload segment. Dec 27 22:55:03.791485131 (535h26m3.662568176s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.791591594 (535h26m3.662674639s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.793523884 (535h26m3.664606939s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.794185644 (535h26m3.665268689s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.795222378 (535h26m3.666305434s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797183563 (535h26m3.668266618s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797315324 (535h26m3.668398379s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.799339678 (535h26m3.670422733s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.800966926 (535h26m3.672049981s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.802493253 (535h26m3.673576298s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.819392662 (535h26m3.690475708s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.823879014 (535h26m3.694962069s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.828981661 (535h26m3.700064716s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.830805173 (535h26m3.701888238s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.832266125 (535h26m3.70334918s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.836751787 (535h26m3.707834832s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:05.507079228 (535h26m5.378162263s) [upload=469598]: Next returned nil segment Dec 27 22:55:05.507093797 (535h26m5.378176842s) [upload=469598]: Waiting for error group managing segments... Dec 27 22:55:05.795763839 (535h26m5.666846884s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.802905938 (535h26m5.673988983s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.803078106 (535h26m5.674161151s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.812284336 (535h26m5.683367381s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.829206028 (535h26m5.700289073s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.835000362 (535h26m5.706083397s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.836389036 (535h26m5.707472071s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.843026765 (535h26m5.7141098s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.859561069 (535h26m5.730644104s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.871635229 (535h26m5.742718274s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.080787678 (535h26m5.951870724s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.094740707 (535h26m5.965823752s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122401899 (535h26m5.993484944s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122423880 (535h26m5.993506925s) [seg_pos={0 0}, upload=469598]: Segment reached optimal threshold of 13 pieces. Dec 27 22:55:06.122791581 (535h26m5.993874616s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection Dec 27 22:55:06.122800467 (535h26m5.993883502s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.122806378 (535h26m5.993889413s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.122825335 (535h26m5.99390838s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123315488 (535h26m5.994398523s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.123323623 (535h26m5.994406668s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123341488 (535h26m5.994424533s) [seg_pos={0 0}, upload=469598]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:55:06.123451447 (535h26m5.994534492s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:55:06.123459001 (535h26m5.994542046s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:55:06.413593572 (535h26m6.284676607s) [upload=469598]: Done uploading segments... Dec 27 22:55:06.413613590 (535h26m6.284696626s) [upload=469598]: Upload finished. err: <nil> Duration: 6.076410907s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [....X............X.X...X.....X...X..X....X.......X.....X....] duration: 9.651185553s trace: https://jaeger.dev.storj.io/trace/63027bab39a48860 FAIL (2025-12-27 22:58:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA) context deadline exceeded ======LOGS====== Dec 27 22:55:03.533152286 (535h26m3.404235331s) [upload=469598]: Starting upload Dec 27 22:55:03.533478477 (535h26m3.404561522s) [upload=469598]: Done starting upload Dec 27 22:55:03.533881645 (535h26m3.40496469s) [upload=469598]: Uploading segments... Dec 27 22:55:03.544825083 (535h26m3.415908128s) [upload=469598]: Got next segment. Inline: false Dec 27 22:55:03.544990097 (535h26m3.416073142s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:55:03.544999244 (535h26m3.41608229s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:55:03.791379841 (535h26m3.662462886s) [seg_pos={0 0}, upload=469598]: Begin upload segment... Dec 27 22:55:03.791445777 (535h26m3.662528812s) [seg_pos={0 0}, upload=469598]: Done begin upload segment. Dec 27 22:55:03.791485131 (535h26m3.662568176s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.791591594 (535h26m3.662674639s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.793523884 (535h26m3.664606939s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.794185644 (535h26m3.665268689s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.795222378 (535h26m3.666305434s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797183563 (535h26m3.668266618s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797315324 (535h26m3.668398379s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.799339678 (535h26m3.670422733s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.800966926 (535h26m3.672049981s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.802493253 (535h26m3.673576298s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.819392662 (535h26m3.690475708s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.823879014 (535h26m3.694962069s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.828981661 (535h26m3.700064716s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.830805173 (535h26m3.701888238s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.832266125 (535h26m3.70334918s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.836751787 (535h26m3.707834832s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:05.507079228 (535h26m5.378162263s) [upload=469598]: Next returned nil segment Dec 27 22:55:05.507093797 (535h26m5.378176842s) [upload=469598]: Waiting for error group managing segments... Dec 27 22:55:05.795763839 (535h26m5.666846884s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.802905938 (535h26m5.673988983s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.803078106 (535h26m5.674161151s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.812284336 (535h26m5.683367381s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.829206028 (535h26m5.700289073s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.835000362 (535h26m5.706083397s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.836389036 (535h26m5.707472071s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.843026765 (535h26m5.7141098s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.859561069 (535h26m5.730644104s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.871635229 (535h26m5.742718274s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.080787678 (535h26m5.951870724s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.094740707 (535h26m5.965823752s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122401899 (535h26m5.993484944s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122423880 (535h26m5.993506925s) [seg_pos={0 0}, upload=469598]: Segment reached optimal threshold of 13 pieces. Dec 27 22:55:06.122791581 (535h26m5.993874616s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection Dec 27 22:55:06.122800467 (535h26m5.993883502s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.122806378 (535h26m5.993889413s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.122825335 (535h26m5.99390838s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123315488 (535h26m5.994398523s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.123323623 (535h26m5.994406668s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123341488 (535h26m5.994424533s) [seg_pos={0 0}, upload=469598]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:55:06.123451447 (535h26m5.994534492s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:55:06.123459001 (535h26m5.994542046s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:55:06.413593572 (535h26m6.284676607s) [upload=469598]: Done uploading segments... Dec 27 22:55:06.413613590 (535h26m6.284696626s) [upload=469598]: Upload finished. err: <nil> Duration: 9.651265084s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.....X............X.X...X.....X...X..X....X.......X.....X...] duration: 5.934181458s trace: https://jaeger.dev.storj.io/trace/63027bab39a48860 FAIL (2025-12-27 22:57:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA) context deadline exceeded ======LOGS====== Dec 27 22:55:03.533152286 (535h26m3.404235331s) [upload=469598]: Starting upload Dec 27 22:55:03.533478477 (535h26m3.404561522s) [upload=469598]: Done starting upload Dec 27 22:55:03.533881645 (535h26m3.40496469s) [upload=469598]: Uploading segments... Dec 27 22:55:03.544825083 (535h26m3.415908128s) [upload=469598]: Got next segment. Inline: false Dec 27 22:55:03.544990097 (535h26m3.416073142s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:55:03.544999244 (535h26m3.41608229s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:55:03.791379841 (535h26m3.662462886s) [seg_pos={0 0}, upload=469598]: Begin upload segment... Dec 27 22:55:03.791445777 (535h26m3.662528812s) [seg_pos={0 0}, upload=469598]: Done begin upload segment. Dec 27 22:55:03.791485131 (535h26m3.662568176s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.791591594 (535h26m3.662674639s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.793523884 (535h26m3.664606939s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.794185644 (535h26m3.665268689s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.795222378 (535h26m3.666305434s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797183563 (535h26m3.668266618s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797315324 (535h26m3.668398379s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.799339678 (535h26m3.670422733s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.800966926 (535h26m3.672049981s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.802493253 (535h26m3.673576298s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.819392662 (535h26m3.690475708s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.823879014 (535h26m3.694962069s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.828981661 (535h26m3.700064716s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.830805173 (535h26m3.701888238s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.832266125 (535h26m3.70334918s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.836751787 (535h26m3.707834832s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:05.507079228 (535h26m5.378162263s) [upload=469598]: Next returned nil segment Dec 27 22:55:05.507093797 (535h26m5.378176842s) [upload=469598]: Waiting for error group managing segments... Dec 27 22:55:05.795763839 (535h26m5.666846884s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.802905938 (535h26m5.673988983s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.803078106 (535h26m5.674161151s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.812284336 (535h26m5.683367381s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.829206028 (535h26m5.700289073s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.835000362 (535h26m5.706083397s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.836389036 (535h26m5.707472071s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.843026765 (535h26m5.7141098s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.859561069 (535h26m5.730644104s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.871635229 (535h26m5.742718274s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.080787678 (535h26m5.951870724s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.094740707 (535h26m5.965823752s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122401899 (535h26m5.993484944s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122423880 (535h26m5.993506925s) [seg_pos={0 0}, upload=469598]: Segment reached optimal threshold of 13 pieces. Dec 27 22:55:06.122791581 (535h26m5.993874616s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection Dec 27 22:55:06.122800467 (535h26m5.993883502s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.122806378 (535h26m5.993889413s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.122825335 (535h26m5.99390838s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123315488 (535h26m5.994398523s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.123323623 (535h26m5.994406668s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123341488 (535h26m5.994424533s) [seg_pos={0 0}, upload=469598]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:55:06.123451447 (535h26m5.994534492s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:55:06.123459001 (535h26m5.994542046s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:55:06.413593572 (535h26m6.284676607s) [upload=469598]: Done uploading segments... Dec 27 22:55:06.413613590 (535h26m6.284696626s) [upload=469598]: Upload finished. err: <nil> Duration: 5.934215543s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [......X............X.X...X.....X...X..X....X.......X.....X..] duration: 26.231038988s trace: https://jaeger.dev.storj.io/trace/63027bab39a48860 FAIL (2025-12-27 22:56:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA) context deadline exceeded ======LOGS====== Dec 27 22:55:03.533152286 (535h26m3.404235331s) [upload=469598]: Starting upload Dec 27 22:55:03.533478477 (535h26m3.404561522s) [upload=469598]: Done starting upload Dec 27 22:55:03.533881645 (535h26m3.40496469s) [upload=469598]: Uploading segments... Dec 27 22:55:03.544825083 (535h26m3.415908128s) [upload=469598]: Got next segment. Inline: false Dec 27 22:55:03.544990097 (535h26m3.416073142s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:55:03.544999244 (535h26m3.41608229s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:55:03.791379841 (535h26m3.662462886s) [seg_pos={0 0}, upload=469598]: Begin upload segment... Dec 27 22:55:03.791445777 (535h26m3.662528812s) [seg_pos={0 0}, upload=469598]: Done begin upload segment. Dec 27 22:55:03.791485131 (535h26m3.662568176s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.791591594 (535h26m3.662674639s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.793523884 (535h26m3.664606939s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.794185644 (535h26m3.665268689s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.795222378 (535h26m3.666305434s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797183563 (535h26m3.668266618s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797315324 (535h26m3.668398379s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.799339678 (535h26m3.670422733s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.800966926 (535h26m3.672049981s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.802493253 (535h26m3.673576298s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.819392662 (535h26m3.690475708s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.823879014 (535h26m3.694962069s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.828981661 (535h26m3.700064716s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.830805173 (535h26m3.701888238s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.832266125 (535h26m3.70334918s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.836751787 (535h26m3.707834832s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:05.507079228 (535h26m5.378162263s) [upload=469598]: Next returned nil segment Dec 27 22:55:05.507093797 (535h26m5.378176842s) [upload=469598]: Waiting for error group managing segments... Dec 27 22:55:05.795763839 (535h26m5.666846884s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.802905938 (535h26m5.673988983s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.803078106 (535h26m5.674161151s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.812284336 (535h26m5.683367381s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.829206028 (535h26m5.700289073s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.835000362 (535h26m5.706083397s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.836389036 (535h26m5.707472071s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.843026765 (535h26m5.7141098s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.859561069 (535h26m5.730644104s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.871635229 (535h26m5.742718274s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.080787678 (535h26m5.951870724s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.094740707 (535h26m5.965823752s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122401899 (535h26m5.993484944s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122423880 (535h26m5.993506925s) [seg_pos={0 0}, upload=469598]: Segment reached optimal threshold of 13 pieces. Dec 27 22:55:06.122791581 (535h26m5.993874616s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection Dec 27 22:55:06.122800467 (535h26m5.993883502s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.122806378 (535h26m5.993889413s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.122825335 (535h26m5.99390838s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123315488 (535h26m5.994398523s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.123323623 (535h26m5.994406668s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123341488 (535h26m5.994424533s) [seg_pos={0 0}, upload=469598]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:55:06.123451447 (535h26m5.994534492s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:55:06.123459001 (535h26m5.994542046s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:55:06.413593572 (535h26m6.284676607s) [upload=469598]: Done uploading segments... Dec 27 22:55:06.413613590 (535h26m6.284696626s) [upload=469598]: Upload finished. err: <nil> Duration: 26.231067591s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.......X............X.X...X.....X...X..X....X.......X.....X.] duration: 7.278868082s trace: https://jaeger.dev.storj.io/trace/63027bab39a48860 FAIL (2025-12-27 22:56:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA) context deadline exceeded ======LOGS====== Dec 27 22:55:03.533152286 (535h26m3.404235331s) [upload=469598]: Starting upload Dec 27 22:55:03.533478477 (535h26m3.404561522s) [upload=469598]: Done starting upload Dec 27 22:55:03.533881645 (535h26m3.40496469s) [upload=469598]: Uploading segments... Dec 27 22:55:03.544825083 (535h26m3.415908128s) [upload=469598]: Got next segment. Inline: false Dec 27 22:55:03.544990097 (535h26m3.416073142s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:55:03.544999244 (535h26m3.41608229s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:55:03.791379841 (535h26m3.662462886s) [seg_pos={0 0}, upload=469598]: Begin upload segment... Dec 27 22:55:03.791445777 (535h26m3.662528812s) [seg_pos={0 0}, upload=469598]: Done begin upload segment. Dec 27 22:55:03.791485131 (535h26m3.662568176s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.791591594 (535h26m3.662674639s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.793523884 (535h26m3.664606939s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.794185644 (535h26m3.665268689s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.795222378 (535h26m3.666305434s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797183563 (535h26m3.668266618s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797315324 (535h26m3.668398379s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.799339678 (535h26m3.670422733s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.800966926 (535h26m3.672049981s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.802493253 (535h26m3.673576298s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.819392662 (535h26m3.690475708s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.823879014 (535h26m3.694962069s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.828981661 (535h26m3.700064716s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.830805173 (535h26m3.701888238s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.832266125 (535h26m3.70334918s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.836751787 (535h26m3.707834832s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:05.507079228 (535h26m5.378162263s) [upload=469598]: Next returned nil segment Dec 27 22:55:05.507093797 (535h26m5.378176842s) [upload=469598]: Waiting for error group managing segments... Dec 27 22:55:05.795763839 (535h26m5.666846884s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.802905938 (535h26m5.673988983s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.803078106 (535h26m5.674161151s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.812284336 (535h26m5.683367381s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.829206028 (535h26m5.700289073s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.835000362 (535h26m5.706083397s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.836389036 (535h26m5.707472071s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.843026765 (535h26m5.7141098s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.859561069 (535h26m5.730644104s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.871635229 (535h26m5.742718274s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.080787678 (535h26m5.951870724s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.094740707 (535h26m5.965823752s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122401899 (535h26m5.993484944s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122423880 (535h26m5.993506925s) [seg_pos={0 0}, upload=469598]: Segment reached optimal threshold of 13 pieces. Dec 27 22:55:06.122791581 (535h26m5.993874616s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection Dec 27 22:55:06.122800467 (535h26m5.993883502s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.122806378 (535h26m5.993889413s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.122825335 (535h26m5.99390838s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123315488 (535h26m5.994398523s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.123323623 (535h26m5.994406668s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123341488 (535h26m5.994424533s) [seg_pos={0 0}, upload=469598]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:55:06.123451447 (535h26m5.994534492s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:55:06.123459001 (535h26m5.994542046s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:55:06.413593572 (535h26m6.284676607s) [upload=469598]: Done uploading segments... Dec 27 22:55:06.413613590 (535h26m6.284696626s) [upload=469598]: Upload finished. err: <nil> Duration: 7.278896597s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [........X............X.X...X.....X...X..X....X.......X.....X] duration: 30.00290716s trace: https://jaeger.dev.storj.io/trace/63027bab39a48860 FAIL (2025-12-27 22:55:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA) context deadline exceeded ======LOGS====== Dec 27 22:55:03.533152286 (535h26m3.404235331s) [upload=469598]: Starting upload Dec 27 22:55:03.533478477 (535h26m3.404561522s) [upload=469598]: Done starting upload Dec 27 22:55:03.533881645 (535h26m3.40496469s) [upload=469598]: Uploading segments... Dec 27 22:55:03.544825083 (535h26m3.415908128s) [upload=469598]: Got next segment. Inline: false Dec 27 22:55:03.544990097 (535h26m3.416073142s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:55:03.544999244 (535h26m3.41608229s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:55:03.791379841 (535h26m3.662462886s) [seg_pos={0 0}, upload=469598]: Begin upload segment... Dec 27 22:55:03.791445777 (535h26m3.662528812s) [seg_pos={0 0}, upload=469598]: Done begin upload segment. Dec 27 22:55:03.791485131 (535h26m3.662568176s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.791591594 (535h26m3.662674639s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.793523884 (535h26m3.664606939s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.794185644 (535h26m3.665268689s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.795222378 (535h26m3.666305434s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797183563 (535h26m3.668266618s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.797315324 (535h26m3.668398379s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.799339678 (535h26m3.670422733s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.800966926 (535h26m3.672049981s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.802493253 (535h26m3.673576298s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.819392662 (535h26m3.690475708s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.823879014 (535h26m3.694962069s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.828981661 (535h26m3.700064716s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.830805173 (535h26m3.701888238s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.832266125 (535h26m3.70334918s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:03.836751787 (535h26m3.707834832s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Uploading piece... Dec 27 22:55:05.507079228 (535h26m5.378162263s) [upload=469598]: Next returned nil segment Dec 27 22:55:05.507093797 (535h26m5.378176842s) [upload=469598]: Waiting for error group managing segments... Dec 27 22:55:05.795763839 (535h26m5.666846884s) [noise=true , address=145.79.227.51:4409 , piece_id=6XEVWXLCTFLJIESKDP37W4R3P4TBXTKQ2LDFFRZOAYZGIGZSLZKQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.802905938 (535h26m5.673988983s) [noise=true , address=145.79.227.46:4601 , piece_id=7OO33SVD6NDFXDV6YDKUK4HU2FRLRNZHP7TFGJBPOCOAEC2HLOCQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.803078106 (535h26m5.674161151s) [noise=true , address=145.79.227.41:4085 , piece_id=HA322QA4MWURSFRMU4TE4GJGVVHVCYFERQFXGI4627FMHXW7YMWQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.812284336 (535h26m5.683367381s) [noise=true , address=145.79.227.32:4836 , piece_id=OO3FM7FF3CC6KTSVVZFI4DHUWFSKMMQLO3QFWYFIRNM6HPHBXHQQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.829206028 (535h26m5.700289073s) [noise=true , address=145.79.227.20:4438 , piece_id=ZFALA3RPED2KQAQI7JN6OXWEZSWCEU26DTH5Z6MFBNGFXFIAS5ZQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.835000362 (535h26m5.706083397s) [noise=true , address=145.79.227.45:4231 , piece_id=WOOYXCPUKXWSU7MGGXKFHUBPDBWMKMH3LOYWZSPSPMXREIX7CGKA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.836389036 (535h26m5.707472071s) [noise=true , address=145.79.227.48:4302 , piece_id=PULETBRQOG3OUH5YVBKQYEUZ55REGGSB4VX262EP7TXNK4UZ4PHQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.843026765 (535h26m5.7141098s) [noise=true , address=145.79.227.25:4618 , piece_id=WN4ADM5ODW2S7ELNI6JWDZKUWKLKV7IRRUYWHC3G4OZT7PJ5VYPQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.859561069 (535h26m5.730644104s) [noise=true , address=145.79.227.52:4447 , piece_id=GP3M5FBH2RRDEL6WYTYK4T5VRYAX6KRDWYPLARDQUR7ZNB6D2J2Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:05.871635229 (535h26m5.742718274s) [noise=true , address=145.79.227.18:4007 , piece_id=VPLIIO2DTFFU6VYIM42DGM2EQZGIOPIKOVXLV7BP65HL7UJ3HT6Q, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.080787678 (535h26m5.951870724s) [noise=true , address=45.91.116.9:4261 , piece_id=R3SVQRCKE4FOICR57NURQS7QRBNNMF3BEFONBZPNNWA4E3NV2S4A, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.094740707 (535h26m5.965823752s) [noise=true , address=45.91.116.7:4009 , piece_id=X6W2YSWYU4KTQ3SMSVJ436M4BW3IZ5SVOOAHAV4Y5RP2OBV5XJJA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122401899 (535h26m5.993484944s) [noise=true , address=45.91.116.13:4155 , piece_id=JOG4KMBGREIAF22OKNZSWDQSYLCAP54AVZGAPTW4STGWATYWQTWA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: <nil> Dec 27 22:55:06.122423880 (535h26m5.993506925s) [seg_pos={0 0}, upload=469598]: Segment reached optimal threshold of 13 pieces. Dec 27 22:55:06.122791581 (535h26m5.993874616s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54316->145.79.227.37:4567: use of closed network connection Dec 27 22:55:06.122800467 (535h26m5.993883502s) [noise=true , address=145.79.227.37:4567 , piece_id=VIBAW5E27MWU6MOW5XM2VB6OC644T5TNB7WGU2UN652POO6JOBSA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.122806378 (535h26m5.993889413s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.122825335 (535h26m5.99390838s) [noise=true , address=77.90.31.11:4553 , piece_id=7CPJ7QUVVXOTPOIS5PRPNGZEBZKR7FFJHDUDC2IV45QQ2UBVODIA, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123315488 (535h26m5.994398523s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 22:55:06.123323623 (535h26m5.994406668s) [noise=true , address=45.91.116.8:4046 , piece_id=6C4GWXGLSLXUZ2U3BVDVCLR44N4PJN7LLVA5WYQZPRTSELBRGGJQ, seg_pos={0 0}, upload=469598]: Long tail cancellation due to optimal threshold Dec 27 22:55:06.123341488 (535h26m5.994424533s) [seg_pos={0 0}, upload=469598]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:55:06.123451447 (535h26m5.994534492s) [upload=469598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:55:06.123459001 (535h26m5.994542046s) [upload=469598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:55:06.413593572 (535h26m6.284676607s) [upload=469598]: Done uploading segments... Dec 27 22:55:06.413613590 (535h26m6.284696626s) [upload=469598]: Upload finished. err: <nil> Duration: 30.002961614s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.........X............X.X...X.....X...X..X....X.......X.....] duration: 28.737628243s trace: https://jaeger.dev.storj.io/trace/249772df51a41f53 FAIL (2025-12-27 22:54:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 50:52.461085904 (535h21m52.332168939s) [upload=469531]: Starting upload Dec 27 22:50:52.461169103 (535h21m52.332252138s) [upload=469531]: Done starting upload Dec 27 22:50:52.461286086 (535h21m52.332369131s) [upload=469531]: Uploading segments... Dec 27 22:50:52.461467611 (535h21m52.332550656s) [upload=469531]: Got next segment. Inline: false Dec 27 22:50:52.461536713 (535h21m52.332619748s) [upload=469531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:50:52.461543065 (535h21m52.3326261s) [upload=469531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:50:52.710875814 (535h21m52.581958849s) [seg_pos={0 0}, upload=469531]: Begin upload segment... Dec 27 22:50:52.710930999 (535h21m52.582014044s) [seg_pos={0 0}, upload=469531]: Done begin upload segment. Dec 27 22:50:52.710999630 (535h21m52.582082675s) [noise=true , address=45.91.116.14:4189 , piece_id=EXGI3IOO5IDKDAKGSN33UG7A4VWIYAWYBY62SRXECD2TRS4CI7WA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.711050677 (535h21m52.582133722s) [noise=true , address=77.90.31.18:4808 , piece_id=EHL6RX4RYTJPBBHMSVPTAF4ZBAUUSVZGKTHIUH7PT7P3NBYK3I4Q, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.711498409 (535h21m52.582581454s) [noise=true , address=45.91.116.12:4226 , piece_id=OVROBHIF4QTM4PPNIYUPUHJ62MQBNP4KS3CLWNAZBZNGPSGI2AFQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.711773042 (535h21m52.582856077s) [noise=true , address=145.79.227.5:4223 , piece_id=YU4RZOZVT3TZ2G6ZJ34UU66WMMNCBGEWZRWJ5L3LEOKL5JPGO3MA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.712024952 (535h21m52.583107987s) [noise=true , address=77.90.31.19:4841 , piece_id=HS4VSO4DZYCYJVDWKYOVM42XMAWR3OSESB2JK5LO7HXSCYRBQSKA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.713023093 (535h21m52.584106128s) [noise=true , address=145.79.227.51:4409 , piece_id=NGNJR2NGQUUMDGFZ6X36YSWAOEIKGQJXS5HVNGMHTAYA6W3C7CLA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.713444104 (535h21m52.584527149s) [noise=true , address=145.79.227.9:4365 , piece_id=J2CZUJ33JOBP4ADRRCOC274VUM4YQHJMJ5ROM7N722V2TU2PQP7Q, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.714709514 (535h21m52.585792549s) [noise=true , address=145.79.227.17:4972 , piece_id=5YAYN2JAI5V34OUZVN6S42NCTBBQYOOCQKGD57SRR4VF7Y34BPZQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.715073497 (535h21m52.586156532s) [noise=true , address=145.79.227.4:4186 , piece_id=RR6DK7YVWGPISFOW5IH6CGLUPY3O4P5RIQDAE7HEUJEOYGJDP3CQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.716633177 (535h21m52.587716212s) [noise=true , address=145.79.227.50:4375 , piece_id=L4DQBISJCPML3YPLZHRTQEEWZVXHRD5OTIRSK7FHGUN7MLG3EH4A, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.717885492 (535h21m52.588968527s) [noise=true , address=145.79.227.37:4566 , piece_id=WJJNKGMY7R5BY7UCIULETMAACLO7MCQ2MMU33YFE23RF6DJVAIKA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.719465961 (535h21m52.590549006s) [noise=true , address=145.79.227.20:4438 , piece_id=S6QSI7NMLEILDBG7UYVXR5X64ZQ4W7T2T5ZABVC7BYABJJOS6FQQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.720765626 (535h21m52.591848661s) [noise=true , address=145.79.227.36:4519 , piece_id=3WPAG53ZS4CXERFZA372HFRS5BHHBG225YEXYX237A6LKXAHF3QQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.721174254 (535h21m52.592257289s) [noise=true , address=45.91.116.5:4298 , piece_id=X4NE2X7CMNWQ6F7CO3GVI6MRKBIZYJCKIAFTCSFARLOPOTHHRSAA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.722066553 (535h21m52.593149598s) [noise=true , address=45.91.116.7:4011 , piece_id=6X7XW5ZYGNLZ5DDQI75IXOFMZ6DI7Z22CO4GF7IVYECNN6445XVQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.723723057 (535h21m52.594806092s) [noise=true , address=145.79.227.38:4979 , piece_id=HGIS6T6QHPNRKXU3SUSN5SRKBJELHXTA4UNYCRTRAZTRDIR2NPJQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:54.424318986 (535h21m54.295402021s) [upload=469531]: Next returned nil segment Dec 27 22:50:54.424335727 (535h21m54.295418792s) [upload=469531]: Waiting for error group managing segments... Dec 27 22:50:54.704253143 (535h21m54.575336178s) [noise=true , address=145.79.227.5:4223 , piece_id=YU4RZOZVT3TZ2G6ZJ34UU66WMMNCBGEWZRWJ5L3LEOKL5JPGO3MA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.713715471 (535h21m54.584798506s) [noise=true , address=145.79.227.50:4375 , piece_id=L4DQBISJCPML3YPLZHRTQEEWZVXHRD5OTIRSK7FHGUN7MLG3EH4A, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.714936987 (535h21m54.586020022s) [noise=true , address=145.79.227.9:4365 , piece_id=J2CZUJ33JOBP4ADRRCOC274VUM4YQHJMJ5ROM7N722V2TU2PQP7Q, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.724954162 (535h21m54.596037197s) [noise=true , address=145.79.227.4:4186 , piece_id=RR6DK7YVWGPISFOW5IH6CGLUPY3O4P5RIQDAE7HEUJEOYGJDP3CQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.733463625 (535h21m54.60454667s) [noise=true , address=145.79.227.37:4566 , piece_id=WJJNKGMY7R5BY7UCIULETMAACLO7MCQ2MMU33YFE23RF6DJVAIKA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.740845331 (535h21m54.611928366s) [noise=true , address=145.79.227.51:4409 , piece_id=NGNJR2NGQUUMDGFZ6X36YSWAOEIKGQJXS5HVNGMHTAYA6W3C7CLA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.750709694 (535h21m54.621792729s) [noise=true , address=145.79.227.20:4438 , piece_id=S6QSI7NMLEILDBG7UYVXR5X64ZQ4W7T2T5ZABVC7BYABJJOS6FQQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.754624639 (535h21m54.625707674s) [noise=true , address=145.79.227.38:4979 , piece_id=HGIS6T6QHPNRKXU3SUSN5SRKBJELHXTA4UNYCRTRAZTRDIR2NPJQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.848487348 (535h21m54.719570383s) [noise=true , address=145.79.227.36:4519 , piece_id=3WPAG53ZS4CXERFZA372HFRS5BHHBG225YEXYX237A6LKXAHF3QQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.903037509 (535h21m54.774120554s) [noise=true , address=77.90.31.19:4841 , piece_id=HS4VSO4DZYCYJVDWKYOVM42XMAWR3OSESB2JK5LO7HXSCYRBQSKA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.967426356 (535h21m54.838509391s) [noise=true , address=45.91.116.5:4298 , piece_id=X4NE2X7CMNWQ6F7CO3GVI6MRKBIZYJCKIAFTCSFARLOPOTHHRSAA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.973908930 (535h21m54.844991975s) [noise=true , address=77.90.31.18:4808 , piece_id=EHL6RX4RYTJPBBHMSVPTAF4ZBAUUSVZGKTHIUH7PT7P3NBYK3I4Q, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:55.010728199 (535h21m54.881811245s) [noise=true , address=45.91.116.14:4189 , piece_id=EXGI3IOO5IDKDAKGSN33UG7A4VWIYAWYBY62SRXECD2TRS4CI7WA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:55.010747706 (535h21m54.881830741s) [seg_pos={0 0}, upload=469531]: Segment reached optimal threshold of 13 pieces. Dec 27 22:50:55.010957275 (535h21m54.88204031s) [noise=true , address=45.91.116.12:4226 , piece_id=OVROBHIF4QTM4PPNIYUPUHJ62MQBNP4KS3CLWNAZBZNGPSGI2AFQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 22:50:55.010964570 (535h21m54.882047625s) [noise=true , address=45.91.116.12:4226 , piece_id=OVROBHIF4QTM4PPNIYUPUHJ62MQBNP4KS3CLWNAZBZNGPSGI2AFQ, seg_pos={0 0}, upload=469531]: Long tail cancellation due to optimal threshold Dec 27 22:50:55.011165852 (535h21m54.882248887s) [noise=true , address=145.79.227.17:4972 , piece_id=5YAYN2JAI5V34OUZVN6S42NCTBBQYOOCQKGD57SRR4VF7Y34BPZQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: write tcp 188.166.225.170:55554->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:55554->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55554->145.79.227.17:4972: use of closed network connection Dec 27 22:50:55.011171995 (535h21m54.88225504s) [noise=true , address=145.79.227.17:4972 , piece_id=5YAYN2JAI5V34OUZVN6S42NCTBBQYOOCQKGD57SRR4VF7Y34BPZQ, seg_pos={0 0}, upload=469531]: Long tail cancellation due to optimal threshold Dec 27 22:50:55.011223212 (535h21m54.882306257s) [noise=true , address=45.91.116.7:4011 , piece_id=6X7XW5ZYGNLZ5DDQI75IXOFMZ6DI7Z22CO4GF7IVYECNN6445XVQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 27 22:50:55.011241487 (535h21m54.882324532s) [noise=true , address=45.91.116.7:4011 , piece_id=6X7XW5ZYGNLZ5DDQI75IXOFMZ6DI7Z22CO4GF7IVYECNN6445XVQ, seg_pos={0 0}, upload=469531]: Long tail cancellation due to optimal threshold Dec 27 22:50:55.011288997 (535h21m54.882372032s) [seg_pos={0 0}, upload=469531]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:50:55.011361084 (535h21m54.882444129s) [upload=469531]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:50:55.011365843 (535h21m54.882448888s) [upload=469531]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:50:55.303924079 (535h21m55.175007115s) [upload=469531]: Done uploading segments... Dec 27 22:50:55.303943547 (535h21m55.175026582s) [upload=469531]: Upload finished. err: <nil> Duration: 28.737656086s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..........X............X.X...X.....X...X..X....X.......X....] duration: 5.888813251s trace: https://jaeger.dev.storj.io/trace/249772df51a41f53 FAIL (2025-12-27 22:53:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 50:52.461085904 (535h21m52.332168939s) [upload=469531]: Starting upload Dec 27 22:50:52.461169103 (535h21m52.332252138s) [upload=469531]: Done starting upload Dec 27 22:50:52.461286086 (535h21m52.332369131s) [upload=469531]: Uploading segments... Dec 27 22:50:52.461467611 (535h21m52.332550656s) [upload=469531]: Got next segment. Inline: false Dec 27 22:50:52.461536713 (535h21m52.332619748s) [upload=469531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:50:52.461543065 (535h21m52.3326261s) [upload=469531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:50:52.710875814 (535h21m52.581958849s) [seg_pos={0 0}, upload=469531]: Begin upload segment... Dec 27 22:50:52.710930999 (535h21m52.582014044s) [seg_pos={0 0}, upload=469531]: Done begin upload segment. Dec 27 22:50:52.710999630 (535h21m52.582082675s) [noise=true , address=45.91.116.14:4189 , piece_id=EXGI3IOO5IDKDAKGSN33UG7A4VWIYAWYBY62SRXECD2TRS4CI7WA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.711050677 (535h21m52.582133722s) [noise=true , address=77.90.31.18:4808 , piece_id=EHL6RX4RYTJPBBHMSVPTAF4ZBAUUSVZGKTHIUH7PT7P3NBYK3I4Q, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.711498409 (535h21m52.582581454s) [noise=true , address=45.91.116.12:4226 , piece_id=OVROBHIF4QTM4PPNIYUPUHJ62MQBNP4KS3CLWNAZBZNGPSGI2AFQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.711773042 (535h21m52.582856077s) [noise=true , address=145.79.227.5:4223 , piece_id=YU4RZOZVT3TZ2G6ZJ34UU66WMMNCBGEWZRWJ5L3LEOKL5JPGO3MA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.712024952 (535h21m52.583107987s) [noise=true , address=77.90.31.19:4841 , piece_id=HS4VSO4DZYCYJVDWKYOVM42XMAWR3OSESB2JK5LO7HXSCYRBQSKA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.713023093 (535h21m52.584106128s) [noise=true , address=145.79.227.51:4409 , piece_id=NGNJR2NGQUUMDGFZ6X36YSWAOEIKGQJXS5HVNGMHTAYA6W3C7CLA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.713444104 (535h21m52.584527149s) [noise=true , address=145.79.227.9:4365 , piece_id=J2CZUJ33JOBP4ADRRCOC274VUM4YQHJMJ5ROM7N722V2TU2PQP7Q, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.714709514 (535h21m52.585792549s) [noise=true , address=145.79.227.17:4972 , piece_id=5YAYN2JAI5V34OUZVN6S42NCTBBQYOOCQKGD57SRR4VF7Y34BPZQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.715073497 (535h21m52.586156532s) [noise=true , address=145.79.227.4:4186 , piece_id=RR6DK7YVWGPISFOW5IH6CGLUPY3O4P5RIQDAE7HEUJEOYGJDP3CQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.716633177 (535h21m52.587716212s) [noise=true , address=145.79.227.50:4375 , piece_id=L4DQBISJCPML3YPLZHRTQEEWZVXHRD5OTIRSK7FHGUN7MLG3EH4A, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.717885492 (535h21m52.588968527s) [noise=true , address=145.79.227.37:4566 , piece_id=WJJNKGMY7R5BY7UCIULETMAACLO7MCQ2MMU33YFE23RF6DJVAIKA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.719465961 (535h21m52.590549006s) [noise=true , address=145.79.227.20:4438 , piece_id=S6QSI7NMLEILDBG7UYVXR5X64ZQ4W7T2T5ZABVC7BYABJJOS6FQQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.720765626 (535h21m52.591848661s) [noise=true , address=145.79.227.36:4519 , piece_id=3WPAG53ZS4CXERFZA372HFRS5BHHBG225YEXYX237A6LKXAHF3QQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.721174254 (535h21m52.592257289s) [noise=true , address=45.91.116.5:4298 , piece_id=X4NE2X7CMNWQ6F7CO3GVI6MRKBIZYJCKIAFTCSFARLOPOTHHRSAA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.722066553 (535h21m52.593149598s) [noise=true , address=45.91.116.7:4011 , piece_id=6X7XW5ZYGNLZ5DDQI75IXOFMZ6DI7Z22CO4GF7IVYECNN6445XVQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.723723057 (535h21m52.594806092s) [noise=true , address=145.79.227.38:4979 , piece_id=HGIS6T6QHPNRKXU3SUSN5SRKBJELHXTA4UNYCRTRAZTRDIR2NPJQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:54.424318986 (535h21m54.295402021s) [upload=469531]: Next returned nil segment Dec 27 22:50:54.424335727 (535h21m54.295418792s) [upload=469531]: Waiting for error group managing segments... Dec 27 22:50:54.704253143 (535h21m54.575336178s) [noise=true , address=145.79.227.5:4223 , piece_id=YU4RZOZVT3TZ2G6ZJ34UU66WMMNCBGEWZRWJ5L3LEOKL5JPGO3MA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.713715471 (535h21m54.584798506s) [noise=true , address=145.79.227.50:4375 , piece_id=L4DQBISJCPML3YPLZHRTQEEWZVXHRD5OTIRSK7FHGUN7MLG3EH4A, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.714936987 (535h21m54.586020022s) [noise=true , address=145.79.227.9:4365 , piece_id=J2CZUJ33JOBP4ADRRCOC274VUM4YQHJMJ5ROM7N722V2TU2PQP7Q, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.724954162 (535h21m54.596037197s) [noise=true , address=145.79.227.4:4186 , piece_id=RR6DK7YVWGPISFOW5IH6CGLUPY3O4P5RIQDAE7HEUJEOYGJDP3CQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.733463625 (535h21m54.60454667s) [noise=true , address=145.79.227.37:4566 , piece_id=WJJNKGMY7R5BY7UCIULETMAACLO7MCQ2MMU33YFE23RF6DJVAIKA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.740845331 (535h21m54.611928366s) [noise=true , address=145.79.227.51:4409 , piece_id=NGNJR2NGQUUMDGFZ6X36YSWAOEIKGQJXS5HVNGMHTAYA6W3C7CLA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.750709694 (535h21m54.621792729s) [noise=true , address=145.79.227.20:4438 , piece_id=S6QSI7NMLEILDBG7UYVXR5X64ZQ4W7T2T5ZABVC7BYABJJOS6FQQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.754624639 (535h21m54.625707674s) [noise=true , address=145.79.227.38:4979 , piece_id=HGIS6T6QHPNRKXU3SUSN5SRKBJELHXTA4UNYCRTRAZTRDIR2NPJQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.848487348 (535h21m54.719570383s) [noise=true , address=145.79.227.36:4519 , piece_id=3WPAG53ZS4CXERFZA372HFRS5BHHBG225YEXYX237A6LKXAHF3QQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.903037509 (535h21m54.774120554s) [noise=true , address=77.90.31.19:4841 , piece_id=HS4VSO4DZYCYJVDWKYOVM42XMAWR3OSESB2JK5LO7HXSCYRBQSKA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.967426356 (535h21m54.838509391s) [noise=true , address=45.91.116.5:4298 , piece_id=X4NE2X7CMNWQ6F7CO3GVI6MRKBIZYJCKIAFTCSFARLOPOTHHRSAA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.973908930 (535h21m54.844991975s) [noise=true , address=77.90.31.18:4808 , piece_id=EHL6RX4RYTJPBBHMSVPTAF4ZBAUUSVZGKTHIUH7PT7P3NBYK3I4Q, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:55.010728199 (535h21m54.881811245s) [noise=true , address=45.91.116.14:4189 , piece_id=EXGI3IOO5IDKDAKGSN33UG7A4VWIYAWYBY62SRXECD2TRS4CI7WA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:55.010747706 (535h21m54.881830741s) [seg_pos={0 0}, upload=469531]: Segment reached optimal threshold of 13 pieces. Dec 27 22:50:55.010957275 (535h21m54.88204031s) [noise=true , address=45.91.116.12:4226 , piece_id=OVROBHIF4QTM4PPNIYUPUHJ62MQBNP4KS3CLWNAZBZNGPSGI2AFQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 22:50:55.010964570 (535h21m54.882047625s) [noise=true , address=45.91.116.12:4226 , piece_id=OVROBHIF4QTM4PPNIYUPUHJ62MQBNP4KS3CLWNAZBZNGPSGI2AFQ, seg_pos={0 0}, upload=469531]: Long tail cancellation due to optimal threshold Dec 27 22:50:55.011165852 (535h21m54.882248887s) [noise=true , address=145.79.227.17:4972 , piece_id=5YAYN2JAI5V34OUZVN6S42NCTBBQYOOCQKGD57SRR4VF7Y34BPZQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: write tcp 188.166.225.170:55554->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:55554->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55554->145.79.227.17:4972: use of closed network connection Dec 27 22:50:55.011171995 (535h21m54.88225504s) [noise=true , address=145.79.227.17:4972 , piece_id=5YAYN2JAI5V34OUZVN6S42NCTBBQYOOCQKGD57SRR4VF7Y34BPZQ, seg_pos={0 0}, upload=469531]: Long tail cancellation due to optimal threshold Dec 27 22:50:55.011223212 (535h21m54.882306257s) [noise=true , address=45.91.116.7:4011 , piece_id=6X7XW5ZYGNLZ5DDQI75IXOFMZ6DI7Z22CO4GF7IVYECNN6445XVQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 27 22:50:55.011241487 (535h21m54.882324532s) [noise=true , address=45.91.116.7:4011 , piece_id=6X7XW5ZYGNLZ5DDQI75IXOFMZ6DI7Z22CO4GF7IVYECNN6445XVQ, seg_pos={0 0}, upload=469531]: Long tail cancellation due to optimal threshold Dec 27 22:50:55.011288997 (535h21m54.882372032s) [seg_pos={0 0}, upload=469531]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:50:55.011361084 (535h21m54.882444129s) [upload=469531]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:50:55.011365843 (535h21m54.882448888s) [upload=469531]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:50:55.303924079 (535h21m55.175007115s) [upload=469531]: Done uploading segments... Dec 27 22:50:55.303943547 (535h21m55.175026582s) [upload=469531]: Upload finished. err: <nil> Duration: 5.888839822s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...........X............X.X...X.....X...X..X....X.......X...] duration: 26.001881424s trace: https://jaeger.dev.storj.io/trace/249772df51a41f53 FAIL (2025-12-27 22:53:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 50:52.461085904 (535h21m52.332168939s) [upload=469531]: Starting upload Dec 27 22:50:52.461169103 (535h21m52.332252138s) [upload=469531]: Done starting upload Dec 27 22:50:52.461286086 (535h21m52.332369131s) [upload=469531]: Uploading segments... Dec 27 22:50:52.461467611 (535h21m52.332550656s) [upload=469531]: Got next segment. Inline: false Dec 27 22:50:52.461536713 (535h21m52.332619748s) [upload=469531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:50:52.461543065 (535h21m52.3326261s) [upload=469531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:50:52.710875814 (535h21m52.581958849s) [seg_pos={0 0}, upload=469531]: Begin upload segment... Dec 27 22:50:52.710930999 (535h21m52.582014044s) [seg_pos={0 0}, upload=469531]: Done begin upload segment. Dec 27 22:50:52.710999630 (535h21m52.582082675s) [noise=true , address=45.91.116.14:4189 , piece_id=EXGI3IOO5IDKDAKGSN33UG7A4VWIYAWYBY62SRXECD2TRS4CI7WA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.711050677 (535h21m52.582133722s) [noise=true , address=77.90.31.18:4808 , piece_id=EHL6RX4RYTJPBBHMSVPTAF4ZBAUUSVZGKTHIUH7PT7P3NBYK3I4Q, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.711498409 (535h21m52.582581454s) [noise=true , address=45.91.116.12:4226 , piece_id=OVROBHIF4QTM4PPNIYUPUHJ62MQBNP4KS3CLWNAZBZNGPSGI2AFQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.711773042 (535h21m52.582856077s) [noise=true , address=145.79.227.5:4223 , piece_id=YU4RZOZVT3TZ2G6ZJ34UU66WMMNCBGEWZRWJ5L3LEOKL5JPGO3MA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.712024952 (535h21m52.583107987s) [noise=true , address=77.90.31.19:4841 , piece_id=HS4VSO4DZYCYJVDWKYOVM42XMAWR3OSESB2JK5LO7HXSCYRBQSKA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.713023093 (535h21m52.584106128s) [noise=true , address=145.79.227.51:4409 , piece_id=NGNJR2NGQUUMDGFZ6X36YSWAOEIKGQJXS5HVNGMHTAYA6W3C7CLA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.713444104 (535h21m52.584527149s) [noise=true , address=145.79.227.9:4365 , piece_id=J2CZUJ33JOBP4ADRRCOC274VUM4YQHJMJ5ROM7N722V2TU2PQP7Q, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.714709514 (535h21m52.585792549s) [noise=true , address=145.79.227.17:4972 , piece_id=5YAYN2JAI5V34OUZVN6S42NCTBBQYOOCQKGD57SRR4VF7Y34BPZQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.715073497 (535h21m52.586156532s) [noise=true , address=145.79.227.4:4186 , piece_id=RR6DK7YVWGPISFOW5IH6CGLUPY3O4P5RIQDAE7HEUJEOYGJDP3CQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.716633177 (535h21m52.587716212s) [noise=true , address=145.79.227.50:4375 , piece_id=L4DQBISJCPML3YPLZHRTQEEWZVXHRD5OTIRSK7FHGUN7MLG3EH4A, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.717885492 (535h21m52.588968527s) [noise=true , address=145.79.227.37:4566 , piece_id=WJJNKGMY7R5BY7UCIULETMAACLO7MCQ2MMU33YFE23RF6DJVAIKA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.719465961 (535h21m52.590549006s) [noise=true , address=145.79.227.20:4438 , piece_id=S6QSI7NMLEILDBG7UYVXR5X64ZQ4W7T2T5ZABVC7BYABJJOS6FQQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.720765626 (535h21m52.591848661s) [noise=true , address=145.79.227.36:4519 , piece_id=3WPAG53ZS4CXERFZA372HFRS5BHHBG225YEXYX237A6LKXAHF3QQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.721174254 (535h21m52.592257289s) [noise=true , address=45.91.116.5:4298 , piece_id=X4NE2X7CMNWQ6F7CO3GVI6MRKBIZYJCKIAFTCSFARLOPOTHHRSAA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.722066553 (535h21m52.593149598s) [noise=true , address=45.91.116.7:4011 , piece_id=6X7XW5ZYGNLZ5DDQI75IXOFMZ6DI7Z22CO4GF7IVYECNN6445XVQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.723723057 (535h21m52.594806092s) [noise=true , address=145.79.227.38:4979 , piece_id=HGIS6T6QHPNRKXU3SUSN5SRKBJELHXTA4UNYCRTRAZTRDIR2NPJQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:54.424318986 (535h21m54.295402021s) [upload=469531]: Next returned nil segment Dec 27 22:50:54.424335727 (535h21m54.295418792s) [upload=469531]: Waiting for error group managing segments... Dec 27 22:50:54.704253143 (535h21m54.575336178s) [noise=true , address=145.79.227.5:4223 , piece_id=YU4RZOZVT3TZ2G6ZJ34UU66WMMNCBGEWZRWJ5L3LEOKL5JPGO3MA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.713715471 (535h21m54.584798506s) [noise=true , address=145.79.227.50:4375 , piece_id=L4DQBISJCPML3YPLZHRTQEEWZVXHRD5OTIRSK7FHGUN7MLG3EH4A, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.714936987 (535h21m54.586020022s) [noise=true , address=145.79.227.9:4365 , piece_id=J2CZUJ33JOBP4ADRRCOC274VUM4YQHJMJ5ROM7N722V2TU2PQP7Q, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.724954162 (535h21m54.596037197s) [noise=true , address=145.79.227.4:4186 , piece_id=RR6DK7YVWGPISFOW5IH6CGLUPY3O4P5RIQDAE7HEUJEOYGJDP3CQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.733463625 (535h21m54.60454667s) [noise=true , address=145.79.227.37:4566 , piece_id=WJJNKGMY7R5BY7UCIULETMAACLO7MCQ2MMU33YFE23RF6DJVAIKA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.740845331 (535h21m54.611928366s) [noise=true , address=145.79.227.51:4409 , piece_id=NGNJR2NGQUUMDGFZ6X36YSWAOEIKGQJXS5HVNGMHTAYA6W3C7CLA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.750709694 (535h21m54.621792729s) [noise=true , address=145.79.227.20:4438 , piece_id=S6QSI7NMLEILDBG7UYVXR5X64ZQ4W7T2T5ZABVC7BYABJJOS6FQQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.754624639 (535h21m54.625707674s) [noise=true , address=145.79.227.38:4979 , piece_id=HGIS6T6QHPNRKXU3SUSN5SRKBJELHXTA4UNYCRTRAZTRDIR2NPJQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.848487348 (535h21m54.719570383s) [noise=true , address=145.79.227.36:4519 , piece_id=3WPAG53ZS4CXERFZA372HFRS5BHHBG225YEXYX237A6LKXAHF3QQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.903037509 (535h21m54.774120554s) [noise=true , address=77.90.31.19:4841 , piece_id=HS4VSO4DZYCYJVDWKYOVM42XMAWR3OSESB2JK5LO7HXSCYRBQSKA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.967426356 (535h21m54.838509391s) [noise=true , address=45.91.116.5:4298 , piece_id=X4NE2X7CMNWQ6F7CO3GVI6MRKBIZYJCKIAFTCSFARLOPOTHHRSAA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.973908930 (535h21m54.844991975s) [noise=true , address=77.90.31.18:4808 , piece_id=EHL6RX4RYTJPBBHMSVPTAF4ZBAUUSVZGKTHIUH7PT7P3NBYK3I4Q, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:55.010728199 (535h21m54.881811245s) [noise=true , address=45.91.116.14:4189 , piece_id=EXGI3IOO5IDKDAKGSN33UG7A4VWIYAWYBY62SRXECD2TRS4CI7WA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:55.010747706 (535h21m54.881830741s) [seg_pos={0 0}, upload=469531]: Segment reached optimal threshold of 13 pieces. Dec 27 22:50:55.010957275 (535h21m54.88204031s) [noise=true , address=45.91.116.12:4226 , piece_id=OVROBHIF4QTM4PPNIYUPUHJ62MQBNP4KS3CLWNAZBZNGPSGI2AFQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 22:50:55.010964570 (535h21m54.882047625s) [noise=true , address=45.91.116.12:4226 , piece_id=OVROBHIF4QTM4PPNIYUPUHJ62MQBNP4KS3CLWNAZBZNGPSGI2AFQ, seg_pos={0 0}, upload=469531]: Long tail cancellation due to optimal threshold Dec 27 22:50:55.011165852 (535h21m54.882248887s) [noise=true , address=145.79.227.17:4972 , piece_id=5YAYN2JAI5V34OUZVN6S42NCTBBQYOOCQKGD57SRR4VF7Y34BPZQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: write tcp 188.166.225.170:55554->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:55554->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55554->145.79.227.17:4972: use of closed network connection Dec 27 22:50:55.011171995 (535h21m54.88225504s) [noise=true , address=145.79.227.17:4972 , piece_id=5YAYN2JAI5V34OUZVN6S42NCTBBQYOOCQKGD57SRR4VF7Y34BPZQ, seg_pos={0 0}, upload=469531]: Long tail cancellation due to optimal threshold Dec 27 22:50:55.011223212 (535h21m54.882306257s) [noise=true , address=45.91.116.7:4011 , piece_id=6X7XW5ZYGNLZ5DDQI75IXOFMZ6DI7Z22CO4GF7IVYECNN6445XVQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 27 22:50:55.011241487 (535h21m54.882324532s) [noise=true , address=45.91.116.7:4011 , piece_id=6X7XW5ZYGNLZ5DDQI75IXOFMZ6DI7Z22CO4GF7IVYECNN6445XVQ, seg_pos={0 0}, upload=469531]: Long tail cancellation due to optimal threshold Dec 27 22:50:55.011288997 (535h21m54.882372032s) [seg_pos={0 0}, upload=469531]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:50:55.011361084 (535h21m54.882444129s) [upload=469531]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:50:55.011365843 (535h21m54.882448888s) [upload=469531]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:50:55.303924079 (535h21m55.175007115s) [upload=469531]: Done uploading segments... Dec 27 22:50:55.303943547 (535h21m55.175026582s) [upload=469531]: Upload finished. err: <nil> Duration: 26.002023724s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [............X............X.X...X.....X...X..X....X.......X..] duration: 5.993791531s trace: https://jaeger.dev.storj.io/trace/249772df51a41f53 FAIL (2025-12-27 22:52:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 50:52.461085904 (535h21m52.332168939s) [upload=469531]: Starting upload Dec 27 22:50:52.461169103 (535h21m52.332252138s) [upload=469531]: Done starting upload Dec 27 22:50:52.461286086 (535h21m52.332369131s) [upload=469531]: Uploading segments... Dec 27 22:50:52.461467611 (535h21m52.332550656s) [upload=469531]: Got next segment. Inline: false Dec 27 22:50:52.461536713 (535h21m52.332619748s) [upload=469531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:50:52.461543065 (535h21m52.3326261s) [upload=469531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:50:52.710875814 (535h21m52.581958849s) [seg_pos={0 0}, upload=469531]: Begin upload segment... Dec 27 22:50:52.710930999 (535h21m52.582014044s) [seg_pos={0 0}, upload=469531]: Done begin upload segment. Dec 27 22:50:52.710999630 (535h21m52.582082675s) [noise=true , address=45.91.116.14:4189 , piece_id=EXGI3IOO5IDKDAKGSN33UG7A4VWIYAWYBY62SRXECD2TRS4CI7WA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.711050677 (535h21m52.582133722s) [noise=true , address=77.90.31.18:4808 , piece_id=EHL6RX4RYTJPBBHMSVPTAF4ZBAUUSVZGKTHIUH7PT7P3NBYK3I4Q, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.711498409 (535h21m52.582581454s) [noise=true , address=45.91.116.12:4226 , piece_id=OVROBHIF4QTM4PPNIYUPUHJ62MQBNP4KS3CLWNAZBZNGPSGI2AFQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.711773042 (535h21m52.582856077s) [noise=true , address=145.79.227.5:4223 , piece_id=YU4RZOZVT3TZ2G6ZJ34UU66WMMNCBGEWZRWJ5L3LEOKL5JPGO3MA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.712024952 (535h21m52.583107987s) [noise=true , address=77.90.31.19:4841 , piece_id=HS4VSO4DZYCYJVDWKYOVM42XMAWR3OSESB2JK5LO7HXSCYRBQSKA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.713023093 (535h21m52.584106128s) [noise=true , address=145.79.227.51:4409 , piece_id=NGNJR2NGQUUMDGFZ6X36YSWAOEIKGQJXS5HVNGMHTAYA6W3C7CLA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.713444104 (535h21m52.584527149s) [noise=true , address=145.79.227.9:4365 , piece_id=J2CZUJ33JOBP4ADRRCOC274VUM4YQHJMJ5ROM7N722V2TU2PQP7Q, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.714709514 (535h21m52.585792549s) [noise=true , address=145.79.227.17:4972 , piece_id=5YAYN2JAI5V34OUZVN6S42NCTBBQYOOCQKGD57SRR4VF7Y34BPZQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.715073497 (535h21m52.586156532s) [noise=true , address=145.79.227.4:4186 , piece_id=RR6DK7YVWGPISFOW5IH6CGLUPY3O4P5RIQDAE7HEUJEOYGJDP3CQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.716633177 (535h21m52.587716212s) [noise=true , address=145.79.227.50:4375 , piece_id=L4DQBISJCPML3YPLZHRTQEEWZVXHRD5OTIRSK7FHGUN7MLG3EH4A, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.717885492 (535h21m52.588968527s) [noise=true , address=145.79.227.37:4566 , piece_id=WJJNKGMY7R5BY7UCIULETMAACLO7MCQ2MMU33YFE23RF6DJVAIKA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.719465961 (535h21m52.590549006s) [noise=true , address=145.79.227.20:4438 , piece_id=S6QSI7NMLEILDBG7UYVXR5X64ZQ4W7T2T5ZABVC7BYABJJOS6FQQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.720765626 (535h21m52.591848661s) [noise=true , address=145.79.227.36:4519 , piece_id=3WPAG53ZS4CXERFZA372HFRS5BHHBG225YEXYX237A6LKXAHF3QQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.721174254 (535h21m52.592257289s) [noise=true , address=45.91.116.5:4298 , piece_id=X4NE2X7CMNWQ6F7CO3GVI6MRKBIZYJCKIAFTCSFARLOPOTHHRSAA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.722066553 (535h21m52.593149598s) [noise=true , address=45.91.116.7:4011 , piece_id=6X7XW5ZYGNLZ5DDQI75IXOFMZ6DI7Z22CO4GF7IVYECNN6445XVQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.723723057 (535h21m52.594806092s) [noise=true , address=145.79.227.38:4979 , piece_id=HGIS6T6QHPNRKXU3SUSN5SRKBJELHXTA4UNYCRTRAZTRDIR2NPJQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:54.424318986 (535h21m54.295402021s) [upload=469531]: Next returned nil segment Dec 27 22:50:54.424335727 (535h21m54.295418792s) [upload=469531]: Waiting for error group managing segments... Dec 27 22:50:54.704253143 (535h21m54.575336178s) [noise=true , address=145.79.227.5:4223 , piece_id=YU4RZOZVT3TZ2G6ZJ34UU66WMMNCBGEWZRWJ5L3LEOKL5JPGO3MA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.713715471 (535h21m54.584798506s) [noise=true , address=145.79.227.50:4375 , piece_id=L4DQBISJCPML3YPLZHRTQEEWZVXHRD5OTIRSK7FHGUN7MLG3EH4A, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.714936987 (535h21m54.586020022s) [noise=true , address=145.79.227.9:4365 , piece_id=J2CZUJ33JOBP4ADRRCOC274VUM4YQHJMJ5ROM7N722V2TU2PQP7Q, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.724954162 (535h21m54.596037197s) [noise=true , address=145.79.227.4:4186 , piece_id=RR6DK7YVWGPISFOW5IH6CGLUPY3O4P5RIQDAE7HEUJEOYGJDP3CQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.733463625 (535h21m54.60454667s) [noise=true , address=145.79.227.37:4566 , piece_id=WJJNKGMY7R5BY7UCIULETMAACLO7MCQ2MMU33YFE23RF6DJVAIKA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.740845331 (535h21m54.611928366s) [noise=true , address=145.79.227.51:4409 , piece_id=NGNJR2NGQUUMDGFZ6X36YSWAOEIKGQJXS5HVNGMHTAYA6W3C7CLA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.750709694 (535h21m54.621792729s) [noise=true , address=145.79.227.20:4438 , piece_id=S6QSI7NMLEILDBG7UYVXR5X64ZQ4W7T2T5ZABVC7BYABJJOS6FQQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.754624639 (535h21m54.625707674s) [noise=true , address=145.79.227.38:4979 , piece_id=HGIS6T6QHPNRKXU3SUSN5SRKBJELHXTA4UNYCRTRAZTRDIR2NPJQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.848487348 (535h21m54.719570383s) [noise=true , address=145.79.227.36:4519 , piece_id=3WPAG53ZS4CXERFZA372HFRS5BHHBG225YEXYX237A6LKXAHF3QQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.903037509 (535h21m54.774120554s) [noise=true , address=77.90.31.19:4841 , piece_id=HS4VSO4DZYCYJVDWKYOVM42XMAWR3OSESB2JK5LO7HXSCYRBQSKA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.967426356 (535h21m54.838509391s) [noise=true , address=45.91.116.5:4298 , piece_id=X4NE2X7CMNWQ6F7CO3GVI6MRKBIZYJCKIAFTCSFARLOPOTHHRSAA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.973908930 (535h21m54.844991975s) [noise=true , address=77.90.31.18:4808 , piece_id=EHL6RX4RYTJPBBHMSVPTAF4ZBAUUSVZGKTHIUH7PT7P3NBYK3I4Q, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:55.010728199 (535h21m54.881811245s) [noise=true , address=45.91.116.14:4189 , piece_id=EXGI3IOO5IDKDAKGSN33UG7A4VWIYAWYBY62SRXECD2TRS4CI7WA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:55.010747706 (535h21m54.881830741s) [seg_pos={0 0}, upload=469531]: Segment reached optimal threshold of 13 pieces. Dec 27 22:50:55.010957275 (535h21m54.88204031s) [noise=true , address=45.91.116.12:4226 , piece_id=OVROBHIF4QTM4PPNIYUPUHJ62MQBNP4KS3CLWNAZBZNGPSGI2AFQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 22:50:55.010964570 (535h21m54.882047625s) [noise=true , address=45.91.116.12:4226 , piece_id=OVROBHIF4QTM4PPNIYUPUHJ62MQBNP4KS3CLWNAZBZNGPSGI2AFQ, seg_pos={0 0}, upload=469531]: Long tail cancellation due to optimal threshold Dec 27 22:50:55.011165852 (535h21m54.882248887s) [noise=true , address=145.79.227.17:4972 , piece_id=5YAYN2JAI5V34OUZVN6S42NCTBBQYOOCQKGD57SRR4VF7Y34BPZQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: write tcp 188.166.225.170:55554->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:55554->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55554->145.79.227.17:4972: use of closed network connection Dec 27 22:50:55.011171995 (535h21m54.88225504s) [noise=true , address=145.79.227.17:4972 , piece_id=5YAYN2JAI5V34OUZVN6S42NCTBBQYOOCQKGD57SRR4VF7Y34BPZQ, seg_pos={0 0}, upload=469531]: Long tail cancellation due to optimal threshold Dec 27 22:50:55.011223212 (535h21m54.882306257s) [noise=true , address=45.91.116.7:4011 , piece_id=6X7XW5ZYGNLZ5DDQI75IXOFMZ6DI7Z22CO4GF7IVYECNN6445XVQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 27 22:50:55.011241487 (535h21m54.882324532s) [noise=true , address=45.91.116.7:4011 , piece_id=6X7XW5ZYGNLZ5DDQI75IXOFMZ6DI7Z22CO4GF7IVYECNN6445XVQ, seg_pos={0 0}, upload=469531]: Long tail cancellation due to optimal threshold Dec 27 22:50:55.011288997 (535h21m54.882372032s) [seg_pos={0 0}, upload=469531]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:50:55.011361084 (535h21m54.882444129s) [upload=469531]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:50:55.011365843 (535h21m54.882448888s) [upload=469531]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:50:55.303924079 (535h21m55.175007115s) [upload=469531]: Done uploading segments... Dec 27 22:50:55.303943547 (535h21m55.175026582s) [upload=469531]: Upload finished. err: <nil> Duration: 5.993818261s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.............X............X.X...X.....X...X..X....X.......X.] duration: 8.891245319s trace: https://jaeger.dev.storj.io/trace/249772df51a41f53 FAIL (2025-12-27 22:51:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 50:52.461085904 (535h21m52.332168939s) [upload=469531]: Starting upload Dec 27 22:50:52.461169103 (535h21m52.332252138s) [upload=469531]: Done starting upload Dec 27 22:50:52.461286086 (535h21m52.332369131s) [upload=469531]: Uploading segments... Dec 27 22:50:52.461467611 (535h21m52.332550656s) [upload=469531]: Got next segment. Inline: false Dec 27 22:50:52.461536713 (535h21m52.332619748s) [upload=469531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:50:52.461543065 (535h21m52.3326261s) [upload=469531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:50:52.710875814 (535h21m52.581958849s) [seg_pos={0 0}, upload=469531]: Begin upload segment... Dec 27 22:50:52.710930999 (535h21m52.582014044s) [seg_pos={0 0}, upload=469531]: Done begin upload segment. Dec 27 22:50:52.710999630 (535h21m52.582082675s) [noise=true , address=45.91.116.14:4189 , piece_id=EXGI3IOO5IDKDAKGSN33UG7A4VWIYAWYBY62SRXECD2TRS4CI7WA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.711050677 (535h21m52.582133722s) [noise=true , address=77.90.31.18:4808 , piece_id=EHL6RX4RYTJPBBHMSVPTAF4ZBAUUSVZGKTHIUH7PT7P3NBYK3I4Q, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.711498409 (535h21m52.582581454s) [noise=true , address=45.91.116.12:4226 , piece_id=OVROBHIF4QTM4PPNIYUPUHJ62MQBNP4KS3CLWNAZBZNGPSGI2AFQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.711773042 (535h21m52.582856077s) [noise=true , address=145.79.227.5:4223 , piece_id=YU4RZOZVT3TZ2G6ZJ34UU66WMMNCBGEWZRWJ5L3LEOKL5JPGO3MA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.712024952 (535h21m52.583107987s) [noise=true , address=77.90.31.19:4841 , piece_id=HS4VSO4DZYCYJVDWKYOVM42XMAWR3OSESB2JK5LO7HXSCYRBQSKA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.713023093 (535h21m52.584106128s) [noise=true , address=145.79.227.51:4409 , piece_id=NGNJR2NGQUUMDGFZ6X36YSWAOEIKGQJXS5HVNGMHTAYA6W3C7CLA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.713444104 (535h21m52.584527149s) [noise=true , address=145.79.227.9:4365 , piece_id=J2CZUJ33JOBP4ADRRCOC274VUM4YQHJMJ5ROM7N722V2TU2PQP7Q, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.714709514 (535h21m52.585792549s) [noise=true , address=145.79.227.17:4972 , piece_id=5YAYN2JAI5V34OUZVN6S42NCTBBQYOOCQKGD57SRR4VF7Y34BPZQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.715073497 (535h21m52.586156532s) [noise=true , address=145.79.227.4:4186 , piece_id=RR6DK7YVWGPISFOW5IH6CGLUPY3O4P5RIQDAE7HEUJEOYGJDP3CQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.716633177 (535h21m52.587716212s) [noise=true , address=145.79.227.50:4375 , piece_id=L4DQBISJCPML3YPLZHRTQEEWZVXHRD5OTIRSK7FHGUN7MLG3EH4A, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.717885492 (535h21m52.588968527s) [noise=true , address=145.79.227.37:4566 , piece_id=WJJNKGMY7R5BY7UCIULETMAACLO7MCQ2MMU33YFE23RF6DJVAIKA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.719465961 (535h21m52.590549006s) [noise=true , address=145.79.227.20:4438 , piece_id=S6QSI7NMLEILDBG7UYVXR5X64ZQ4W7T2T5ZABVC7BYABJJOS6FQQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.720765626 (535h21m52.591848661s) [noise=true , address=145.79.227.36:4519 , piece_id=3WPAG53ZS4CXERFZA372HFRS5BHHBG225YEXYX237A6LKXAHF3QQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.721174254 (535h21m52.592257289s) [noise=true , address=45.91.116.5:4298 , piece_id=X4NE2X7CMNWQ6F7CO3GVI6MRKBIZYJCKIAFTCSFARLOPOTHHRSAA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.722066553 (535h21m52.593149598s) [noise=true , address=45.91.116.7:4011 , piece_id=6X7XW5ZYGNLZ5DDQI75IXOFMZ6DI7Z22CO4GF7IVYECNN6445XVQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.723723057 (535h21m52.594806092s) [noise=true , address=145.79.227.38:4979 , piece_id=HGIS6T6QHPNRKXU3SUSN5SRKBJELHXTA4UNYCRTRAZTRDIR2NPJQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:54.424318986 (535h21m54.295402021s) [upload=469531]: Next returned nil segment Dec 27 22:50:54.424335727 (535h21m54.295418792s) [upload=469531]: Waiting for error group managing segments... Dec 27 22:50:54.704253143 (535h21m54.575336178s) [noise=true , address=145.79.227.5:4223 , piece_id=YU4RZOZVT3TZ2G6ZJ34UU66WMMNCBGEWZRWJ5L3LEOKL5JPGO3MA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.713715471 (535h21m54.584798506s) [noise=true , address=145.79.227.50:4375 , piece_id=L4DQBISJCPML3YPLZHRTQEEWZVXHRD5OTIRSK7FHGUN7MLG3EH4A, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.714936987 (535h21m54.586020022s) [noise=true , address=145.79.227.9:4365 , piece_id=J2CZUJ33JOBP4ADRRCOC274VUM4YQHJMJ5ROM7N722V2TU2PQP7Q, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.724954162 (535h21m54.596037197s) [noise=true , address=145.79.227.4:4186 , piece_id=RR6DK7YVWGPISFOW5IH6CGLUPY3O4P5RIQDAE7HEUJEOYGJDP3CQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.733463625 (535h21m54.60454667s) [noise=true , address=145.79.227.37:4566 , piece_id=WJJNKGMY7R5BY7UCIULETMAACLO7MCQ2MMU33YFE23RF6DJVAIKA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.740845331 (535h21m54.611928366s) [noise=true , address=145.79.227.51:4409 , piece_id=NGNJR2NGQUUMDGFZ6X36YSWAOEIKGQJXS5HVNGMHTAYA6W3C7CLA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.750709694 (535h21m54.621792729s) [noise=true , address=145.79.227.20:4438 , piece_id=S6QSI7NMLEILDBG7UYVXR5X64ZQ4W7T2T5ZABVC7BYABJJOS6FQQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.754624639 (535h21m54.625707674s) [noise=true , address=145.79.227.38:4979 , piece_id=HGIS6T6QHPNRKXU3SUSN5SRKBJELHXTA4UNYCRTRAZTRDIR2NPJQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.848487348 (535h21m54.719570383s) [noise=true , address=145.79.227.36:4519 , piece_id=3WPAG53ZS4CXERFZA372HFRS5BHHBG225YEXYX237A6LKXAHF3QQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.903037509 (535h21m54.774120554s) [noise=true , address=77.90.31.19:4841 , piece_id=HS4VSO4DZYCYJVDWKYOVM42XMAWR3OSESB2JK5LO7HXSCYRBQSKA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.967426356 (535h21m54.838509391s) [noise=true , address=45.91.116.5:4298 , piece_id=X4NE2X7CMNWQ6F7CO3GVI6MRKBIZYJCKIAFTCSFARLOPOTHHRSAA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.973908930 (535h21m54.844991975s) [noise=true , address=77.90.31.18:4808 , piece_id=EHL6RX4RYTJPBBHMSVPTAF4ZBAUUSVZGKTHIUH7PT7P3NBYK3I4Q, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:55.010728199 (535h21m54.881811245s) [noise=true , address=45.91.116.14:4189 , piece_id=EXGI3IOO5IDKDAKGSN33UG7A4VWIYAWYBY62SRXECD2TRS4CI7WA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:55.010747706 (535h21m54.881830741s) [seg_pos={0 0}, upload=469531]: Segment reached optimal threshold of 13 pieces. Dec 27 22:50:55.010957275 (535h21m54.88204031s) [noise=true , address=45.91.116.12:4226 , piece_id=OVROBHIF4QTM4PPNIYUPUHJ62MQBNP4KS3CLWNAZBZNGPSGI2AFQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 22:50:55.010964570 (535h21m54.882047625s) [noise=true , address=45.91.116.12:4226 , piece_id=OVROBHIF4QTM4PPNIYUPUHJ62MQBNP4KS3CLWNAZBZNGPSGI2AFQ, seg_pos={0 0}, upload=469531]: Long tail cancellation due to optimal threshold Dec 27 22:50:55.011165852 (535h21m54.882248887s) [noise=true , address=145.79.227.17:4972 , piece_id=5YAYN2JAI5V34OUZVN6S42NCTBBQYOOCQKGD57SRR4VF7Y34BPZQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: write tcp 188.166.225.170:55554->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:55554->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55554->145.79.227.17:4972: use of closed network connection Dec 27 22:50:55.011171995 (535h21m54.88225504s) [noise=true , address=145.79.227.17:4972 , piece_id=5YAYN2JAI5V34OUZVN6S42NCTBBQYOOCQKGD57SRR4VF7Y34BPZQ, seg_pos={0 0}, upload=469531]: Long tail cancellation due to optimal threshold Dec 27 22:50:55.011223212 (535h21m54.882306257s) [noise=true , address=45.91.116.7:4011 , piece_id=6X7XW5ZYGNLZ5DDQI75IXOFMZ6DI7Z22CO4GF7IVYECNN6445XVQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 27 22:50:55.011241487 (535h21m54.882324532s) [noise=true , address=45.91.116.7:4011 , piece_id=6X7XW5ZYGNLZ5DDQI75IXOFMZ6DI7Z22CO4GF7IVYECNN6445XVQ, seg_pos={0 0}, upload=469531]: Long tail cancellation due to optimal threshold Dec 27 22:50:55.011288997 (535h21m54.882372032s) [seg_pos={0 0}, upload=469531]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:50:55.011361084 (535h21m54.882444129s) [upload=469531]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:50:55.011365843 (535h21m54.882448888s) [upload=469531]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:50:55.303924079 (535h21m55.175007115s) [upload=469531]: Done uploading segments... Dec 27 22:50:55.303943547 (535h21m55.175026582s) [upload=469531]: Upload finished. err: <nil> Duration: 8.891274476s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..............X............X.X...X.....X...X..X....X.......X] duration: 30.002740624s trace: https://jaeger.dev.storj.io/trace/249772df51a41f53 FAIL (2025-12-27 22:51:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 50:52.461085904 (535h21m52.332168939s) [upload=469531]: Starting upload Dec 27 22:50:52.461169103 (535h21m52.332252138s) [upload=469531]: Done starting upload Dec 27 22:50:52.461286086 (535h21m52.332369131s) [upload=469531]: Uploading segments... Dec 27 22:50:52.461467611 (535h21m52.332550656s) [upload=469531]: Got next segment. Inline: false Dec 27 22:50:52.461536713 (535h21m52.332619748s) [upload=469531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:50:52.461543065 (535h21m52.3326261s) [upload=469531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:50:52.710875814 (535h21m52.581958849s) [seg_pos={0 0}, upload=469531]: Begin upload segment... Dec 27 22:50:52.710930999 (535h21m52.582014044s) [seg_pos={0 0}, upload=469531]: Done begin upload segment. Dec 27 22:50:52.710999630 (535h21m52.582082675s) [noise=true , address=45.91.116.14:4189 , piece_id=EXGI3IOO5IDKDAKGSN33UG7A4VWIYAWYBY62SRXECD2TRS4CI7WA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.711050677 (535h21m52.582133722s) [noise=true , address=77.90.31.18:4808 , piece_id=EHL6RX4RYTJPBBHMSVPTAF4ZBAUUSVZGKTHIUH7PT7P3NBYK3I4Q, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.711498409 (535h21m52.582581454s) [noise=true , address=45.91.116.12:4226 , piece_id=OVROBHIF4QTM4PPNIYUPUHJ62MQBNP4KS3CLWNAZBZNGPSGI2AFQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.711773042 (535h21m52.582856077s) [noise=true , address=145.79.227.5:4223 , piece_id=YU4RZOZVT3TZ2G6ZJ34UU66WMMNCBGEWZRWJ5L3LEOKL5JPGO3MA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.712024952 (535h21m52.583107987s) [noise=true , address=77.90.31.19:4841 , piece_id=HS4VSO4DZYCYJVDWKYOVM42XMAWR3OSESB2JK5LO7HXSCYRBQSKA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.713023093 (535h21m52.584106128s) [noise=true , address=145.79.227.51:4409 , piece_id=NGNJR2NGQUUMDGFZ6X36YSWAOEIKGQJXS5HVNGMHTAYA6W3C7CLA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.713444104 (535h21m52.584527149s) [noise=true , address=145.79.227.9:4365 , piece_id=J2CZUJ33JOBP4ADRRCOC274VUM4YQHJMJ5ROM7N722V2TU2PQP7Q, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.714709514 (535h21m52.585792549s) [noise=true , address=145.79.227.17:4972 , piece_id=5YAYN2JAI5V34OUZVN6S42NCTBBQYOOCQKGD57SRR4VF7Y34BPZQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.715073497 (535h21m52.586156532s) [noise=true , address=145.79.227.4:4186 , piece_id=RR6DK7YVWGPISFOW5IH6CGLUPY3O4P5RIQDAE7HEUJEOYGJDP3CQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.716633177 (535h21m52.587716212s) [noise=true , address=145.79.227.50:4375 , piece_id=L4DQBISJCPML3YPLZHRTQEEWZVXHRD5OTIRSK7FHGUN7MLG3EH4A, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.717885492 (535h21m52.588968527s) [noise=true , address=145.79.227.37:4566 , piece_id=WJJNKGMY7R5BY7UCIULETMAACLO7MCQ2MMU33YFE23RF6DJVAIKA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.719465961 (535h21m52.590549006s) [noise=true , address=145.79.227.20:4438 , piece_id=S6QSI7NMLEILDBG7UYVXR5X64ZQ4W7T2T5ZABVC7BYABJJOS6FQQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.720765626 (535h21m52.591848661s) [noise=true , address=145.79.227.36:4519 , piece_id=3WPAG53ZS4CXERFZA372HFRS5BHHBG225YEXYX237A6LKXAHF3QQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.721174254 (535h21m52.592257289s) [noise=true , address=45.91.116.5:4298 , piece_id=X4NE2X7CMNWQ6F7CO3GVI6MRKBIZYJCKIAFTCSFARLOPOTHHRSAA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.722066553 (535h21m52.593149598s) [noise=true , address=45.91.116.7:4011 , piece_id=6X7XW5ZYGNLZ5DDQI75IXOFMZ6DI7Z22CO4GF7IVYECNN6445XVQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.723723057 (535h21m52.594806092s) [noise=true , address=145.79.227.38:4979 , piece_id=HGIS6T6QHPNRKXU3SUSN5SRKBJELHXTA4UNYCRTRAZTRDIR2NPJQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:54.424318986 (535h21m54.295402021s) [upload=469531]: Next returned nil segment Dec 27 22:50:54.424335727 (535h21m54.295418792s) [upload=469531]: Waiting for error group managing segments... Dec 27 22:50:54.704253143 (535h21m54.575336178s) [noise=true , address=145.79.227.5:4223 , piece_id=YU4RZOZVT3TZ2G6ZJ34UU66WMMNCBGEWZRWJ5L3LEOKL5JPGO3MA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.713715471 (535h21m54.584798506s) [noise=true , address=145.79.227.50:4375 , piece_id=L4DQBISJCPML3YPLZHRTQEEWZVXHRD5OTIRSK7FHGUN7MLG3EH4A, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.714936987 (535h21m54.586020022s) [noise=true , address=145.79.227.9:4365 , piece_id=J2CZUJ33JOBP4ADRRCOC274VUM4YQHJMJ5ROM7N722V2TU2PQP7Q, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.724954162 (535h21m54.596037197s) [noise=true , address=145.79.227.4:4186 , piece_id=RR6DK7YVWGPISFOW5IH6CGLUPY3O4P5RIQDAE7HEUJEOYGJDP3CQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.733463625 (535h21m54.60454667s) [noise=true , address=145.79.227.37:4566 , piece_id=WJJNKGMY7R5BY7UCIULETMAACLO7MCQ2MMU33YFE23RF6DJVAIKA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.740845331 (535h21m54.611928366s) [noise=true , address=145.79.227.51:4409 , piece_id=NGNJR2NGQUUMDGFZ6X36YSWAOEIKGQJXS5HVNGMHTAYA6W3C7CLA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.750709694 (535h21m54.621792729s) [noise=true , address=145.79.227.20:4438 , piece_id=S6QSI7NMLEILDBG7UYVXR5X64ZQ4W7T2T5ZABVC7BYABJJOS6FQQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.754624639 (535h21m54.625707674s) [noise=true , address=145.79.227.38:4979 , piece_id=HGIS6T6QHPNRKXU3SUSN5SRKBJELHXTA4UNYCRTRAZTRDIR2NPJQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.848487348 (535h21m54.719570383s) [noise=true , address=145.79.227.36:4519 , piece_id=3WPAG53ZS4CXERFZA372HFRS5BHHBG225YEXYX237A6LKXAHF3QQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.903037509 (535h21m54.774120554s) [noise=true , address=77.90.31.19:4841 , piece_id=HS4VSO4DZYCYJVDWKYOVM42XMAWR3OSESB2JK5LO7HXSCYRBQSKA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.967426356 (535h21m54.838509391s) [noise=true , address=45.91.116.5:4298 , piece_id=X4NE2X7CMNWQ6F7CO3GVI6MRKBIZYJCKIAFTCSFARLOPOTHHRSAA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.973908930 (535h21m54.844991975s) [noise=true , address=77.90.31.18:4808 , piece_id=EHL6RX4RYTJPBBHMSVPTAF4ZBAUUSVZGKTHIUH7PT7P3NBYK3I4Q, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:55.010728199 (535h21m54.881811245s) [noise=true , address=45.91.116.14:4189 , piece_id=EXGI3IOO5IDKDAKGSN33UG7A4VWIYAWYBY62SRXECD2TRS4CI7WA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:55.010747706 (535h21m54.881830741s) [seg_pos={0 0}, upload=469531]: Segment reached optimal threshold of 13 pieces. Dec 27 22:50:55.010957275 (535h21m54.88204031s) [noise=true , address=45.91.116.12:4226 , piece_id=OVROBHIF4QTM4PPNIYUPUHJ62MQBNP4KS3CLWNAZBZNGPSGI2AFQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 22:50:55.010964570 (535h21m54.882047625s) [noise=true , address=45.91.116.12:4226 , piece_id=OVROBHIF4QTM4PPNIYUPUHJ62MQBNP4KS3CLWNAZBZNGPSGI2AFQ, seg_pos={0 0}, upload=469531]: Long tail cancellation due to optimal threshold Dec 27 22:50:55.011165852 (535h21m54.882248887s) [noise=true , address=145.79.227.17:4972 , piece_id=5YAYN2JAI5V34OUZVN6S42NCTBBQYOOCQKGD57SRR4VF7Y34BPZQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: write tcp 188.166.225.170:55554->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:55554->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55554->145.79.227.17:4972: use of closed network connection Dec 27 22:50:55.011171995 (535h21m54.88225504s) [noise=true , address=145.79.227.17:4972 , piece_id=5YAYN2JAI5V34OUZVN6S42NCTBBQYOOCQKGD57SRR4VF7Y34BPZQ, seg_pos={0 0}, upload=469531]: Long tail cancellation due to optimal threshold Dec 27 22:50:55.011223212 (535h21m54.882306257s) [noise=true , address=45.91.116.7:4011 , piece_id=6X7XW5ZYGNLZ5DDQI75IXOFMZ6DI7Z22CO4GF7IVYECNN6445XVQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 27 22:50:55.011241487 (535h21m54.882324532s) [noise=true , address=45.91.116.7:4011 , piece_id=6X7XW5ZYGNLZ5DDQI75IXOFMZ6DI7Z22CO4GF7IVYECNN6445XVQ, seg_pos={0 0}, upload=469531]: Long tail cancellation due to optimal threshold Dec 27 22:50:55.011288997 (535h21m54.882372032s) [seg_pos={0 0}, upload=469531]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:50:55.011361084 (535h21m54.882444129s) [upload=469531]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:50:55.011365843 (535h21m54.882448888s) [upload=469531]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:50:55.303924079 (535h21m55.175007115s) [upload=469531]: Done uploading segments... Dec 27 22:50:55.303943547 (535h21m55.175026582s) [upload=469531]: Upload finished. err: <nil> Duration: 30.002766273s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...............X............X.X...X.....X...X..X....X.......] duration: 24.896782519s trace: https://jaeger.dev.storj.io/trace/67e5b65c1836be39 FAIL (2025-12-27 22:50:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: ZSUJHALWAQOHVMTXFW54BB6RF7Z3TVGBUYMAKZIUP4KHVZE2YSTA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 6ZI3SEX5K2F32GL3G6TZ5KZPMWOCLYQGA63JCSIUU334WFQQ3TOQ) context deadline exceeded ======LOGS====== 16m4.620323326s) [upload=469438]: Uploading segments... Dec 27 22:45:04.749488463 (535h16m4.620571508s) [upload=469438]: Got next segment. Inline: false Dec 27 22:45:04.749529381 (535h16m4.620612436s) [upload=469438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:45:04.749536094 (535h16m4.620619149s) [upload=469438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:45:05.000717426 (535h16m4.871800501s) [seg_pos={0 0}, upload=469438]: Begin upload segment... Dec 27 22:45:05.000782259 (535h16m4.871865304s) [seg_pos={0 0}, upload=469438]: Done begin upload segment. Dec 27 22:45:05.000821734 (535h16m4.871904789s) [noise=true , address=77.90.31.10:4518 , piece_id=SPJXKTKKCAWKWVJHK34YDRQL7BC4I4GRY4HZYUBL7WZPAT5GYGTA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.001222127 (535h16m4.872305182s) [noise=true , address=45.91.116.8:4047 , piece_id=OIQP4IXTAED2DLC2X6R6TMBK546W7TXXVFVGRCV3WGDCH65T3FQA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.002353952 (535h16m4.873436997s) [noise=true , address=145.79.227.47:4266 , piece_id=DCQJDVWX7NEWG5R6JHRIUQBEMGZJLNNSBMPKMQPT3L6VOJEWK3EQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.002883460 (535h16m4.873966515s) [noise=true , address=145.79.227.24:4582 , piece_id=ZJQ5GMGRKHLAUV3DZ4LYIUDP7CDJ4C4TTYIB5SK5LGECHEW2ZVSQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.002930680 (535h16m4.874013736s) [noise=true , address=145.79.227.5:4223 , piece_id=WNS3IGWVQAGLWPGEJR7474PEUBVKUWG7ISF3P73TXYE3MNCLECLA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.003303590 (535h16m4.874386645s) [noise=true , address=145.79.227.49:4338 , piece_id=UPBP4WKNJB4PG5REJGTAAQEVIRLUVCQHJUV43OTXKT6DGJZGUEHQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.003723670 (535h16m4.874806715s) [noise=true , address=77.90.31.14:4661 , piece_id=ZSUJHALWAQOHVMTXFW54BB6RF7Z3TVGBUYMAKZIUP4KHVZE2YSTA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.004172565 (535h16m4.87525561s) [noise=true , address=145.79.227.37:4567 , piece_id=QHZOH2IPBXQWSBJHBJ4EBJCW5JHHQRURJM4GODY3NP2AOUFWWR5Q, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.004194817 (535h16m4.875277872s) [noise=true , address=145.79.227.42:4122 , piece_id=ALADJYXJQ4TQTCFKQIN3CHH3GTU72ZB7G3WXX74LIPZN4TD3TD6A, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.004633323 (535h16m4.875716378s) [noise=true , address=145.79.227.9:4365 , piece_id=K2IZ3KP7NM6ZU4BZ6RPEO6LVCTYIM33APM2UWJHPNXGMIWVJ6CXA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.005010059 (535h16m4.876093104s) [noise=true , address=45.91.116.5:4297 , piece_id=6ZI3SEX5K2F32GL3G6TZ5KZPMWOCLYQGA63JCSIUU334WFQQ3TOQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.006339952 (535h16m4.877422997s) [noise=true , address=145.79.227.45:4231 , piece_id=IVFX654533G6O4WXZFAG4ONLB5LIOIVEC6CWE4S3QELFDO5KHDEQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.007234004 (535h16m4.878317049s) [noise=true , address=145.79.227.39:4013 , piece_id=VX4DUSCEWUKC5HTCUVUNQEKOSPR4RJVTPOU7YBSWBPQRTGAZ6RNA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.011044781 (535h16m4.882127826s) [noise=true , address=145.79.227.11:4731 , piece_id=6PL34PORDMFYGTF7KU4ZAHRLO7JU66BFD55FEH23EYKUQRHWTERA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.012865718 (535h16m4.883948763s) [noise=true , address=45.91.116.11:4117 , piece_id=P2HLL4ATCVG2QIOKBJVGO2PFJQCLYA5GLURFGPHPEN4JYRJ5S5LQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.014688018 (535h16m4.885771063s) [noise=true , address=145.79.227.38:4977 , piece_id=ZM7RPMQIVMGTXLO55VGICXWNIEVMUEOZGWSY7HHF3OYUPUSJWEKA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:06.624796567 (535h16m6.495879612s) [upload=469438]: Next returned nil segment Dec 27 22:45:06.624814311 (535h16m6.495897356s) [upload=469438]: Waiting for error group managing segments... Dec 27 22:45:07.009031135 (535h16m6.88011418s) [noise=true , address=145.79.227.11:4731 , piece_id=6PL34PORDMFYGTF7KU4ZAHRLO7JU66BFD55FEH23EYKUQRHWTERA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.012820972 (535h16m6.883904017s) [noise=true , address=145.79.227.24:4582 , piece_id=ZJQ5GMGRKHLAUV3DZ4LYIUDP7CDJ4C4TTYIB5SK5LGECHEW2ZVSQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.019694149 (535h16m6.890777184s) [noise=true , address=145.79.227.39:4013 , piece_id=VX4DUSCEWUKC5HTCUVUNQEKOSPR4RJVTPOU7YBSWBPQRTGAZ6RNA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.020947285 (535h16m6.89203033s) [noise=true , address=145.79.227.9:4365 , piece_id=K2IZ3KP7NM6ZU4BZ6RPEO6LVCTYIM33APM2UWJHPNXGMIWVJ6CXA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.026693517 (535h16m6.897776552s) [noise=true , address=145.79.227.42:4122 , piece_id=ALADJYXJQ4TQTCFKQIN3CHH3GTU72ZB7G3WXX74LIPZN4TD3TD6A, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.027854889 (535h16m6.898937924s) [noise=true , address=145.79.227.5:4223 , piece_id=WNS3IGWVQAGLWPGEJR7474PEUBVKUWG7ISF3P73TXYE3MNCLECLA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.028395879 (535h16m6.899478924s) [noise=true , address=145.79.227.38:4977 , piece_id=ZM7RPMQIVMGTXLO55VGICXWNIEVMUEOZGWSY7HHF3OYUPUSJWEKA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.032625914 (535h16m6.903708949s) [noise=true , address=145.79.227.45:4231 , piece_id=IVFX654533G6O4WXZFAG4ONLB5LIOIVEC6CWE4S3QELFDO5KHDEQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.048007063 (535h16m6.919090098s) [noise=true , address=145.79.227.47:4266 , piece_id=DCQJDVWX7NEWG5R6JHRIUQBEMGZJLNNSBMPKMQPT3L6VOJEWK3EQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.167310827 (535h16m7.038393872s) [noise=true , address=77.90.31.14:4661 , piece_id=ZSUJHALWAQOHVMTXFW54BB6RF7Z3TVGBUYMAKZIUP4KHVZE2YSTA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.221778913 (535h16m7.092861948s) [noise=true , address=45.91.116.5:4297 , piece_id=6ZI3SEX5K2F32GL3G6TZ5KZPMWOCLYQGA63JCSIUU334WFQQ3TOQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.244653280 (535h16m7.115736325s) [noise=true , address=45.91.116.11:4117 , piece_id=P2HLL4ATCVG2QIOKBJVGO2PFJQCLYA5GLURFGPHPEN4JYRJ5S5LQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.250513118 (535h16m7.121596153s) [noise=true , address=45.91.116.8:4047 , piece_id=OIQP4IXTAED2DLC2X6R6TMBK546W7TXXVFVGRCV3WGDCH65T3FQA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.250536132 (535h16m7.121619177s) [seg_pos={0 0}, upload=469438]: Segment reached optimal threshold of 13 pieces. Dec 27 22:45:07.250729120 (535h16m7.121812165s) [noise=true , address=145.79.227.37:4567 , piece_id=QHZOH2IPBXQWSBJHBJ4EBJCW5JHHQRURJM4GODY3NP2AOUFWWR5Q, seg_pos={0 0}, upload=469438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:54026->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:54026->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54026->145.79.227.37:4567: use of closed network connection Dec 27 22:45:07.250735572 (535h16m7.121818617s) [noise=true , address=145.79.227.37:4567 , piece_id=QHZOH2IPBXQWSBJHBJ4EBJCW5JHHQRURJM4GODY3NP2AOUFWWR5Q, seg_pos={0 0}, upload=469438]: Long tail cancellation due to optimal threshold Dec 27 22:45:07.250951833 (535h16m7.122034879s) [noise=true , address=77.90.31.10:4518 , piece_id=SPJXKTKKCAWKWVJHK34YDRQL7BC4I4GRY4HZYUBL7WZPAT5GYGTA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:58332->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:58332->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58332->77.90.31.10:4518: use of closed network connection Dec 27 22:45:07.250961772 (535h16m7.122044807s) [noise=true , address=77.90.31.10:4518 , piece_id=SPJXKTKKCAWKWVJHK34YDRQL7BC4I4GRY4HZYUBL7WZPAT5GYGTA, seg_pos={0 0}, upload=469438]: Long tail cancellation due to optimal threshold Dec 27 22:45:07.251122168 (535h16m7.122205203s) [noise=true , address=145.79.227.49:4338 , piece_id=UPBP4WKNJB4PG5REJGTAAQEVIRLUVCQHJUV43OTXKT6DGJZGUEHQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 27 22:45:07.251129642 (535h16m7.122212687s) [noise=true , address=145.79.227.49:4338 , piece_id=UPBP4WKNJB4PG5REJGTAAQEVIRLUVCQHJUV43OTXKT6DGJZGUEHQ, seg_pos={0 0}, upload=469438]: Long tail cancellation due to optimal threshold Dec 27 22:45:07.251146795 (535h16m7.12222983s) [seg_pos={0 0}, upload=469438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:45:07.251218642 (535h16m7.122301677s) [upload=469438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:45:07.251223621 (535h16m7.122306666s) [upload=469438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:45:07.559197465 (535h16m7.430280501s) [upload=469438]: Done uploading segments... Dec 27 22:45:07.559217354 (535h16m7.430300389s) [upload=469438]: Upload finished. err: <nil> Duration: 24.896810803s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [................X............X.X...X.....X...X..X....X......] duration: 6.331413153s trace: https://jaeger.dev.storj.io/trace/67e5b65c1836be39 FAIL (2025-12-27 22:49:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: ZSUJHALWAQOHVMTXFW54BB6RF7Z3TVGBUYMAKZIUP4KHVZE2YSTA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 6ZI3SEX5K2F32GL3G6TZ5KZPMWOCLYQGA63JCSIUU334WFQQ3TOQ) context deadline exceeded ======LOGS====== 16m4.620323326s) [upload=469438]: Uploading segments... Dec 27 22:45:04.749488463 (535h16m4.620571508s) [upload=469438]: Got next segment. Inline: false Dec 27 22:45:04.749529381 (535h16m4.620612436s) [upload=469438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:45:04.749536094 (535h16m4.620619149s) [upload=469438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:45:05.000717426 (535h16m4.871800501s) [seg_pos={0 0}, upload=469438]: Begin upload segment... Dec 27 22:45:05.000782259 (535h16m4.871865304s) [seg_pos={0 0}, upload=469438]: Done begin upload segment. Dec 27 22:45:05.000821734 (535h16m4.871904789s) [noise=true , address=77.90.31.10:4518 , piece_id=SPJXKTKKCAWKWVJHK34YDRQL7BC4I4GRY4HZYUBL7WZPAT5GYGTA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.001222127 (535h16m4.872305182s) [noise=true , address=45.91.116.8:4047 , piece_id=OIQP4IXTAED2DLC2X6R6TMBK546W7TXXVFVGRCV3WGDCH65T3FQA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.002353952 (535h16m4.873436997s) [noise=true , address=145.79.227.47:4266 , piece_id=DCQJDVWX7NEWG5R6JHRIUQBEMGZJLNNSBMPKMQPT3L6VOJEWK3EQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.002883460 (535h16m4.873966515s) [noise=true , address=145.79.227.24:4582 , piece_id=ZJQ5GMGRKHLAUV3DZ4LYIUDP7CDJ4C4TTYIB5SK5LGECHEW2ZVSQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.002930680 (535h16m4.874013736s) [noise=true , address=145.79.227.5:4223 , piece_id=WNS3IGWVQAGLWPGEJR7474PEUBVKUWG7ISF3P73TXYE3MNCLECLA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.003303590 (535h16m4.874386645s) [noise=true , address=145.79.227.49:4338 , piece_id=UPBP4WKNJB4PG5REJGTAAQEVIRLUVCQHJUV43OTXKT6DGJZGUEHQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.003723670 (535h16m4.874806715s) [noise=true , address=77.90.31.14:4661 , piece_id=ZSUJHALWAQOHVMTXFW54BB6RF7Z3TVGBUYMAKZIUP4KHVZE2YSTA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.004172565 (535h16m4.87525561s) [noise=true , address=145.79.227.37:4567 , piece_id=QHZOH2IPBXQWSBJHBJ4EBJCW5JHHQRURJM4GODY3NP2AOUFWWR5Q, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.004194817 (535h16m4.875277872s) [noise=true , address=145.79.227.42:4122 , piece_id=ALADJYXJQ4TQTCFKQIN3CHH3GTU72ZB7G3WXX74LIPZN4TD3TD6A, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.004633323 (535h16m4.875716378s) [noise=true , address=145.79.227.9:4365 , piece_id=K2IZ3KP7NM6ZU4BZ6RPEO6LVCTYIM33APM2UWJHPNXGMIWVJ6CXA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.005010059 (535h16m4.876093104s) [noise=true , address=45.91.116.5:4297 , piece_id=6ZI3SEX5K2F32GL3G6TZ5KZPMWOCLYQGA63JCSIUU334WFQQ3TOQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.006339952 (535h16m4.877422997s) [noise=true , address=145.79.227.45:4231 , piece_id=IVFX654533G6O4WXZFAG4ONLB5LIOIVEC6CWE4S3QELFDO5KHDEQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.007234004 (535h16m4.878317049s) [noise=true , address=145.79.227.39:4013 , piece_id=VX4DUSCEWUKC5HTCUVUNQEKOSPR4RJVTPOU7YBSWBPQRTGAZ6RNA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.011044781 (535h16m4.882127826s) [noise=true , address=145.79.227.11:4731 , piece_id=6PL34PORDMFYGTF7KU4ZAHRLO7JU66BFD55FEH23EYKUQRHWTERA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.012865718 (535h16m4.883948763s) [noise=true , address=45.91.116.11:4117 , piece_id=P2HLL4ATCVG2QIOKBJVGO2PFJQCLYA5GLURFGPHPEN4JYRJ5S5LQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.014688018 (535h16m4.885771063s) [noise=true , address=145.79.227.38:4977 , piece_id=ZM7RPMQIVMGTXLO55VGICXWNIEVMUEOZGWSY7HHF3OYUPUSJWEKA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:06.624796567 (535h16m6.495879612s) [upload=469438]: Next returned nil segment Dec 27 22:45:06.624814311 (535h16m6.495897356s) [upload=469438]: Waiting for error group managing segments... Dec 27 22:45:07.009031135 (535h16m6.88011418s) [noise=true , address=145.79.227.11:4731 , piece_id=6PL34PORDMFYGTF7KU4ZAHRLO7JU66BFD55FEH23EYKUQRHWTERA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.012820972 (535h16m6.883904017s) [noise=true , address=145.79.227.24:4582 , piece_id=ZJQ5GMGRKHLAUV3DZ4LYIUDP7CDJ4C4TTYIB5SK5LGECHEW2ZVSQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.019694149 (535h16m6.890777184s) [noise=true , address=145.79.227.39:4013 , piece_id=VX4DUSCEWUKC5HTCUVUNQEKOSPR4RJVTPOU7YBSWBPQRTGAZ6RNA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.020947285 (535h16m6.89203033s) [noise=true , address=145.79.227.9:4365 , piece_id=K2IZ3KP7NM6ZU4BZ6RPEO6LVCTYIM33APM2UWJHPNXGMIWVJ6CXA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.026693517 (535h16m6.897776552s) [noise=true , address=145.79.227.42:4122 , piece_id=ALADJYXJQ4TQTCFKQIN3CHH3GTU72ZB7G3WXX74LIPZN4TD3TD6A, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.027854889 (535h16m6.898937924s) [noise=true , address=145.79.227.5:4223 , piece_id=WNS3IGWVQAGLWPGEJR7474PEUBVKUWG7ISF3P73TXYE3MNCLECLA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.028395879 (535h16m6.899478924s) [noise=true , address=145.79.227.38:4977 , piece_id=ZM7RPMQIVMGTXLO55VGICXWNIEVMUEOZGWSY7HHF3OYUPUSJWEKA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.032625914 (535h16m6.903708949s) [noise=true , address=145.79.227.45:4231 , piece_id=IVFX654533G6O4WXZFAG4ONLB5LIOIVEC6CWE4S3QELFDO5KHDEQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.048007063 (535h16m6.919090098s) [noise=true , address=145.79.227.47:4266 , piece_id=DCQJDVWX7NEWG5R6JHRIUQBEMGZJLNNSBMPKMQPT3L6VOJEWK3EQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.167310827 (535h16m7.038393872s) [noise=true , address=77.90.31.14:4661 , piece_id=ZSUJHALWAQOHVMTXFW54BB6RF7Z3TVGBUYMAKZIUP4KHVZE2YSTA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.221778913 (535h16m7.092861948s) [noise=true , address=45.91.116.5:4297 , piece_id=6ZI3SEX5K2F32GL3G6TZ5KZPMWOCLYQGA63JCSIUU334WFQQ3TOQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.244653280 (535h16m7.115736325s) [noise=true , address=45.91.116.11:4117 , piece_id=P2HLL4ATCVG2QIOKBJVGO2PFJQCLYA5GLURFGPHPEN4JYRJ5S5LQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.250513118 (535h16m7.121596153s) [noise=true , address=45.91.116.8:4047 , piece_id=OIQP4IXTAED2DLC2X6R6TMBK546W7TXXVFVGRCV3WGDCH65T3FQA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.250536132 (535h16m7.121619177s) [seg_pos={0 0}, upload=469438]: Segment reached optimal threshold of 13 pieces. Dec 27 22:45:07.250729120 (535h16m7.121812165s) [noise=true , address=145.79.227.37:4567 , piece_id=QHZOH2IPBXQWSBJHBJ4EBJCW5JHHQRURJM4GODY3NP2AOUFWWR5Q, seg_pos={0 0}, upload=469438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:54026->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:54026->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54026->145.79.227.37:4567: use of closed network connection Dec 27 22:45:07.250735572 (535h16m7.121818617s) [noise=true , address=145.79.227.37:4567 , piece_id=QHZOH2IPBXQWSBJHBJ4EBJCW5JHHQRURJM4GODY3NP2AOUFWWR5Q, seg_pos={0 0}, upload=469438]: Long tail cancellation due to optimal threshold Dec 27 22:45:07.250951833 (535h16m7.122034879s) [noise=true , address=77.90.31.10:4518 , piece_id=SPJXKTKKCAWKWVJHK34YDRQL7BC4I4GRY4HZYUBL7WZPAT5GYGTA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:58332->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:58332->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58332->77.90.31.10:4518: use of closed network connection Dec 27 22:45:07.250961772 (535h16m7.122044807s) [noise=true , address=77.90.31.10:4518 , piece_id=SPJXKTKKCAWKWVJHK34YDRQL7BC4I4GRY4HZYUBL7WZPAT5GYGTA, seg_pos={0 0}, upload=469438]: Long tail cancellation due to optimal threshold Dec 27 22:45:07.251122168 (535h16m7.122205203s) [noise=true , address=145.79.227.49:4338 , piece_id=UPBP4WKNJB4PG5REJGTAAQEVIRLUVCQHJUV43OTXKT6DGJZGUEHQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 27 22:45:07.251129642 (535h16m7.122212687s) [noise=true , address=145.79.227.49:4338 , piece_id=UPBP4WKNJB4PG5REJGTAAQEVIRLUVCQHJUV43OTXKT6DGJZGUEHQ, seg_pos={0 0}, upload=469438]: Long tail cancellation due to optimal threshold Dec 27 22:45:07.251146795 (535h16m7.12222983s) [seg_pos={0 0}, upload=469438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:45:07.251218642 (535h16m7.122301677s) [upload=469438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:45:07.251223621 (535h16m7.122306666s) [upload=469438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:45:07.559197465 (535h16m7.430280501s) [upload=469438]: Done uploading segments... Dec 27 22:45:07.559217354 (535h16m7.430300389s) [upload=469438]: Upload finished. err: <nil> Duration: 6.331442438s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.................X............X.X...X.....X...X..X....X.....] duration: 5.942168619s trace: https://jaeger.dev.storj.io/trace/67e5b65c1836be39 FAIL (2025-12-27 22:48:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: ZSUJHALWAQOHVMTXFW54BB6RF7Z3TVGBUYMAKZIUP4KHVZE2YSTA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 6ZI3SEX5K2F32GL3G6TZ5KZPMWOCLYQGA63JCSIUU334WFQQ3TOQ) context deadline exceeded ======LOGS====== 16m4.620323326s) [upload=469438]: Uploading segments... Dec 27 22:45:04.749488463 (535h16m4.620571508s) [upload=469438]: Got next segment. Inline: false Dec 27 22:45:04.749529381 (535h16m4.620612436s) [upload=469438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:45:04.749536094 (535h16m4.620619149s) [upload=469438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:45:05.000717426 (535h16m4.871800501s) [seg_pos={0 0}, upload=469438]: Begin upload segment... Dec 27 22:45:05.000782259 (535h16m4.871865304s) [seg_pos={0 0}, upload=469438]: Done begin upload segment. Dec 27 22:45:05.000821734 (535h16m4.871904789s) [noise=true , address=77.90.31.10:4518 , piece_id=SPJXKTKKCAWKWVJHK34YDRQL7BC4I4GRY4HZYUBL7WZPAT5GYGTA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.001222127 (535h16m4.872305182s) [noise=true , address=45.91.116.8:4047 , piece_id=OIQP4IXTAED2DLC2X6R6TMBK546W7TXXVFVGRCV3WGDCH65T3FQA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.002353952 (535h16m4.873436997s) [noise=true , address=145.79.227.47:4266 , piece_id=DCQJDVWX7NEWG5R6JHRIUQBEMGZJLNNSBMPKMQPT3L6VOJEWK3EQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.002883460 (535h16m4.873966515s) [noise=true , address=145.79.227.24:4582 , piece_id=ZJQ5GMGRKHLAUV3DZ4LYIUDP7CDJ4C4TTYIB5SK5LGECHEW2ZVSQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.002930680 (535h16m4.874013736s) [noise=true , address=145.79.227.5:4223 , piece_id=WNS3IGWVQAGLWPGEJR7474PEUBVKUWG7ISF3P73TXYE3MNCLECLA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.003303590 (535h16m4.874386645s) [noise=true , address=145.79.227.49:4338 , piece_id=UPBP4WKNJB4PG5REJGTAAQEVIRLUVCQHJUV43OTXKT6DGJZGUEHQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.003723670 (535h16m4.874806715s) [noise=true , address=77.90.31.14:4661 , piece_id=ZSUJHALWAQOHVMTXFW54BB6RF7Z3TVGBUYMAKZIUP4KHVZE2YSTA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.004172565 (535h16m4.87525561s) [noise=true , address=145.79.227.37:4567 , piece_id=QHZOH2IPBXQWSBJHBJ4EBJCW5JHHQRURJM4GODY3NP2AOUFWWR5Q, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.004194817 (535h16m4.875277872s) [noise=true , address=145.79.227.42:4122 , piece_id=ALADJYXJQ4TQTCFKQIN3CHH3GTU72ZB7G3WXX74LIPZN4TD3TD6A, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.004633323 (535h16m4.875716378s) [noise=true , address=145.79.227.9:4365 , piece_id=K2IZ3KP7NM6ZU4BZ6RPEO6LVCTYIM33APM2UWJHPNXGMIWVJ6CXA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.005010059 (535h16m4.876093104s) [noise=true , address=45.91.116.5:4297 , piece_id=6ZI3SEX5K2F32GL3G6TZ5KZPMWOCLYQGA63JCSIUU334WFQQ3TOQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.006339952 (535h16m4.877422997s) [noise=true , address=145.79.227.45:4231 , piece_id=IVFX654533G6O4WXZFAG4ONLB5LIOIVEC6CWE4S3QELFDO5KHDEQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.007234004 (535h16m4.878317049s) [noise=true , address=145.79.227.39:4013 , piece_id=VX4DUSCEWUKC5HTCUVUNQEKOSPR4RJVTPOU7YBSWBPQRTGAZ6RNA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.011044781 (535h16m4.882127826s) [noise=true , address=145.79.227.11:4731 , piece_id=6PL34PORDMFYGTF7KU4ZAHRLO7JU66BFD55FEH23EYKUQRHWTERA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.012865718 (535h16m4.883948763s) [noise=true , address=45.91.116.11:4117 , piece_id=P2HLL4ATCVG2QIOKBJVGO2PFJQCLYA5GLURFGPHPEN4JYRJ5S5LQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.014688018 (535h16m4.885771063s) [noise=true , address=145.79.227.38:4977 , piece_id=ZM7RPMQIVMGTXLO55VGICXWNIEVMUEOZGWSY7HHF3OYUPUSJWEKA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:06.624796567 (535h16m6.495879612s) [upload=469438]: Next returned nil segment Dec 27 22:45:06.624814311 (535h16m6.495897356s) [upload=469438]: Waiting for error group managing segments... Dec 27 22:45:07.009031135 (535h16m6.88011418s) [noise=true , address=145.79.227.11:4731 , piece_id=6PL34PORDMFYGTF7KU4ZAHRLO7JU66BFD55FEH23EYKUQRHWTERA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.012820972 (535h16m6.883904017s) [noise=true , address=145.79.227.24:4582 , piece_id=ZJQ5GMGRKHLAUV3DZ4LYIUDP7CDJ4C4TTYIB5SK5LGECHEW2ZVSQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.019694149 (535h16m6.890777184s) [noise=true , address=145.79.227.39:4013 , piece_id=VX4DUSCEWUKC5HTCUVUNQEKOSPR4RJVTPOU7YBSWBPQRTGAZ6RNA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.020947285 (535h16m6.89203033s) [noise=true , address=145.79.227.9:4365 , piece_id=K2IZ3KP7NM6ZU4BZ6RPEO6LVCTYIM33APM2UWJHPNXGMIWVJ6CXA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.026693517 (535h16m6.897776552s) [noise=true , address=145.79.227.42:4122 , piece_id=ALADJYXJQ4TQTCFKQIN3CHH3GTU72ZB7G3WXX74LIPZN4TD3TD6A, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.027854889 (535h16m6.898937924s) [noise=true , address=145.79.227.5:4223 , piece_id=WNS3IGWVQAGLWPGEJR7474PEUBVKUWG7ISF3P73TXYE3MNCLECLA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.028395879 (535h16m6.899478924s) [noise=true , address=145.79.227.38:4977 , piece_id=ZM7RPMQIVMGTXLO55VGICXWNIEVMUEOZGWSY7HHF3OYUPUSJWEKA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.032625914 (535h16m6.903708949s) [noise=true , address=145.79.227.45:4231 , piece_id=IVFX654533G6O4WXZFAG4ONLB5LIOIVEC6CWE4S3QELFDO5KHDEQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.048007063 (535h16m6.919090098s) [noise=true , address=145.79.227.47:4266 , piece_id=DCQJDVWX7NEWG5R6JHRIUQBEMGZJLNNSBMPKMQPT3L6VOJEWK3EQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.167310827 (535h16m7.038393872s) [noise=true , address=77.90.31.14:4661 , piece_id=ZSUJHALWAQOHVMTXFW54BB6RF7Z3TVGBUYMAKZIUP4KHVZE2YSTA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.221778913 (535h16m7.092861948s) [noise=true , address=45.91.116.5:4297 , piece_id=6ZI3SEX5K2F32GL3G6TZ5KZPMWOCLYQGA63JCSIUU334WFQQ3TOQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.244653280 (535h16m7.115736325s) [noise=true , address=45.91.116.11:4117 , piece_id=P2HLL4ATCVG2QIOKBJVGO2PFJQCLYA5GLURFGPHPEN4JYRJ5S5LQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.250513118 (535h16m7.121596153s) [noise=true , address=45.91.116.8:4047 , piece_id=OIQP4IXTAED2DLC2X6R6TMBK546W7TXXVFVGRCV3WGDCH65T3FQA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.250536132 (535h16m7.121619177s) [seg_pos={0 0}, upload=469438]: Segment reached optimal threshold of 13 pieces. Dec 27 22:45:07.250729120 (535h16m7.121812165s) [noise=true , address=145.79.227.37:4567 , piece_id=QHZOH2IPBXQWSBJHBJ4EBJCW5JHHQRURJM4GODY3NP2AOUFWWR5Q, seg_pos={0 0}, upload=469438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:54026->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:54026->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54026->145.79.227.37:4567: use of closed network connection Dec 27 22:45:07.250735572 (535h16m7.121818617s) [noise=true , address=145.79.227.37:4567 , piece_id=QHZOH2IPBXQWSBJHBJ4EBJCW5JHHQRURJM4GODY3NP2AOUFWWR5Q, seg_pos={0 0}, upload=469438]: Long tail cancellation due to optimal threshold Dec 27 22:45:07.250951833 (535h16m7.122034879s) [noise=true , address=77.90.31.10:4518 , piece_id=SPJXKTKKCAWKWVJHK34YDRQL7BC4I4GRY4HZYUBL7WZPAT5GYGTA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:58332->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:58332->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58332->77.90.31.10:4518: use of closed network connection Dec 27 22:45:07.250961772 (535h16m7.122044807s) [noise=true , address=77.90.31.10:4518 , piece_id=SPJXKTKKCAWKWVJHK34YDRQL7BC4I4GRY4HZYUBL7WZPAT5GYGTA, seg_pos={0 0}, upload=469438]: Long tail cancellation due to optimal threshold Dec 27 22:45:07.251122168 (535h16m7.122205203s) [noise=true , address=145.79.227.49:4338 , piece_id=UPBP4WKNJB4PG5REJGTAAQEVIRLUVCQHJUV43OTXKT6DGJZGUEHQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 27 22:45:07.251129642 (535h16m7.122212687s) [noise=true , address=145.79.227.49:4338 , piece_id=UPBP4WKNJB4PG5REJGTAAQEVIRLUVCQHJUV43OTXKT6DGJZGUEHQ, seg_pos={0 0}, upload=469438]: Long tail cancellation due to optimal threshold Dec 27 22:45:07.251146795 (535h16m7.12222983s) [seg_pos={0 0}, upload=469438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:45:07.251218642 (535h16m7.122301677s) [upload=469438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:45:07.251223621 (535h16m7.122306666s) [upload=469438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:45:07.559197465 (535h16m7.430280501s) [upload=469438]: Done uploading segments... Dec 27 22:45:07.559217354 (535h16m7.430300389s) [upload=469438]: Upload finished. err: <nil> Duration: 5.94218534s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..................X............X.X...X.....X...X..X....X....] duration: 6.358478357s trace: https://jaeger.dev.storj.io/trace/67e5b65c1836be39 FAIL (2025-12-27 22:47:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: ZSUJHALWAQOHVMTXFW54BB6RF7Z3TVGBUYMAKZIUP4KHVZE2YSTA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 6ZI3SEX5K2F32GL3G6TZ5KZPMWOCLYQGA63JCSIUU334WFQQ3TOQ) context deadline exceeded ======LOGS====== 16m4.620323326s) [upload=469438]: Uploading segments... Dec 27 22:45:04.749488463 (535h16m4.620571508s) [upload=469438]: Got next segment. Inline: false Dec 27 22:45:04.749529381 (535h16m4.620612436s) [upload=469438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:45:04.749536094 (535h16m4.620619149s) [upload=469438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:45:05.000717426 (535h16m4.871800501s) [seg_pos={0 0}, upload=469438]: Begin upload segment... Dec 27 22:45:05.000782259 (535h16m4.871865304s) [seg_pos={0 0}, upload=469438]: Done begin upload segment. Dec 27 22:45:05.000821734 (535h16m4.871904789s) [noise=true , address=77.90.31.10:4518 , piece_id=SPJXKTKKCAWKWVJHK34YDRQL7BC4I4GRY4HZYUBL7WZPAT5GYGTA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.001222127 (535h16m4.872305182s) [noise=true , address=45.91.116.8:4047 , piece_id=OIQP4IXTAED2DLC2X6R6TMBK546W7TXXVFVGRCV3WGDCH65T3FQA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.002353952 (535h16m4.873436997s) [noise=true , address=145.79.227.47:4266 , piece_id=DCQJDVWX7NEWG5R6JHRIUQBEMGZJLNNSBMPKMQPT3L6VOJEWK3EQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.002883460 (535h16m4.873966515s) [noise=true , address=145.79.227.24:4582 , piece_id=ZJQ5GMGRKHLAUV3DZ4LYIUDP7CDJ4C4TTYIB5SK5LGECHEW2ZVSQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.002930680 (535h16m4.874013736s) [noise=true , address=145.79.227.5:4223 , piece_id=WNS3IGWVQAGLWPGEJR7474PEUBVKUWG7ISF3P73TXYE3MNCLECLA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.003303590 (535h16m4.874386645s) [noise=true , address=145.79.227.49:4338 , piece_id=UPBP4WKNJB4PG5REJGTAAQEVIRLUVCQHJUV43OTXKT6DGJZGUEHQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.003723670 (535h16m4.874806715s) [noise=true , address=77.90.31.14:4661 , piece_id=ZSUJHALWAQOHVMTXFW54BB6RF7Z3TVGBUYMAKZIUP4KHVZE2YSTA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.004172565 (535h16m4.87525561s) [noise=true , address=145.79.227.37:4567 , piece_id=QHZOH2IPBXQWSBJHBJ4EBJCW5JHHQRURJM4GODY3NP2AOUFWWR5Q, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.004194817 (535h16m4.875277872s) [noise=true , address=145.79.227.42:4122 , piece_id=ALADJYXJQ4TQTCFKQIN3CHH3GTU72ZB7G3WXX74LIPZN4TD3TD6A, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.004633323 (535h16m4.875716378s) [noise=true , address=145.79.227.9:4365 , piece_id=K2IZ3KP7NM6ZU4BZ6RPEO6LVCTYIM33APM2UWJHPNXGMIWVJ6CXA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.005010059 (535h16m4.876093104s) [noise=true , address=45.91.116.5:4297 , piece_id=6ZI3SEX5K2F32GL3G6TZ5KZPMWOCLYQGA63JCSIUU334WFQQ3TOQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.006339952 (535h16m4.877422997s) [noise=true , address=145.79.227.45:4231 , piece_id=IVFX654533G6O4WXZFAG4ONLB5LIOIVEC6CWE4S3QELFDO5KHDEQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.007234004 (535h16m4.878317049s) [noise=true , address=145.79.227.39:4013 , piece_id=VX4DUSCEWUKC5HTCUVUNQEKOSPR4RJVTPOU7YBSWBPQRTGAZ6RNA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.011044781 (535h16m4.882127826s) [noise=true , address=145.79.227.11:4731 , piece_id=6PL34PORDMFYGTF7KU4ZAHRLO7JU66BFD55FEH23EYKUQRHWTERA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.012865718 (535h16m4.883948763s) [noise=true , address=45.91.116.11:4117 , piece_id=P2HLL4ATCVG2QIOKBJVGO2PFJQCLYA5GLURFGPHPEN4JYRJ5S5LQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.014688018 (535h16m4.885771063s) [noise=true , address=145.79.227.38:4977 , piece_id=ZM7RPMQIVMGTXLO55VGICXWNIEVMUEOZGWSY7HHF3OYUPUSJWEKA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:06.624796567 (535h16m6.495879612s) [upload=469438]: Next returned nil segment Dec 27 22:45:06.624814311 (535h16m6.495897356s) [upload=469438]: Waiting for error group managing segments... Dec 27 22:45:07.009031135 (535h16m6.88011418s) [noise=true , address=145.79.227.11:4731 , piece_id=6PL34PORDMFYGTF7KU4ZAHRLO7JU66BFD55FEH23EYKUQRHWTERA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.012820972 (535h16m6.883904017s) [noise=true , address=145.79.227.24:4582 , piece_id=ZJQ5GMGRKHLAUV3DZ4LYIUDP7CDJ4C4TTYIB5SK5LGECHEW2ZVSQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.019694149 (535h16m6.890777184s) [noise=true , address=145.79.227.39:4013 , piece_id=VX4DUSCEWUKC5HTCUVUNQEKOSPR4RJVTPOU7YBSWBPQRTGAZ6RNA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.020947285 (535h16m6.89203033s) [noise=true , address=145.79.227.9:4365 , piece_id=K2IZ3KP7NM6ZU4BZ6RPEO6LVCTYIM33APM2UWJHPNXGMIWVJ6CXA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.026693517 (535h16m6.897776552s) [noise=true , address=145.79.227.42:4122 , piece_id=ALADJYXJQ4TQTCFKQIN3CHH3GTU72ZB7G3WXX74LIPZN4TD3TD6A, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.027854889 (535h16m6.898937924s) [noise=true , address=145.79.227.5:4223 , piece_id=WNS3IGWVQAGLWPGEJR7474PEUBVKUWG7ISF3P73TXYE3MNCLECLA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.028395879 (535h16m6.899478924s) [noise=true , address=145.79.227.38:4977 , piece_id=ZM7RPMQIVMGTXLO55VGICXWNIEVMUEOZGWSY7HHF3OYUPUSJWEKA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.032625914 (535h16m6.903708949s) [noise=true , address=145.79.227.45:4231 , piece_id=IVFX654533G6O4WXZFAG4ONLB5LIOIVEC6CWE4S3QELFDO5KHDEQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.048007063 (535h16m6.919090098s) [noise=true , address=145.79.227.47:4266 , piece_id=DCQJDVWX7NEWG5R6JHRIUQBEMGZJLNNSBMPKMQPT3L6VOJEWK3EQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.167310827 (535h16m7.038393872s) [noise=true , address=77.90.31.14:4661 , piece_id=ZSUJHALWAQOHVMTXFW54BB6RF7Z3TVGBUYMAKZIUP4KHVZE2YSTA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.221778913 (535h16m7.092861948s) [noise=true , address=45.91.116.5:4297 , piece_id=6ZI3SEX5K2F32GL3G6TZ5KZPMWOCLYQGA63JCSIUU334WFQQ3TOQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.244653280 (535h16m7.115736325s) [noise=true , address=45.91.116.11:4117 , piece_id=P2HLL4ATCVG2QIOKBJVGO2PFJQCLYA5GLURFGPHPEN4JYRJ5S5LQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.250513118 (535h16m7.121596153s) [noise=true , address=45.91.116.8:4047 , piece_id=OIQP4IXTAED2DLC2X6R6TMBK546W7TXXVFVGRCV3WGDCH65T3FQA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.250536132 (535h16m7.121619177s) [seg_pos={0 0}, upload=469438]: Segment reached optimal threshold of 13 pieces. Dec 27 22:45:07.250729120 (535h16m7.121812165s) [noise=true , address=145.79.227.37:4567 , piece_id=QHZOH2IPBXQWSBJHBJ4EBJCW5JHHQRURJM4GODY3NP2AOUFWWR5Q, seg_pos={0 0}, upload=469438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:54026->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:54026->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54026->145.79.227.37:4567: use of closed network connection Dec 27 22:45:07.250735572 (535h16m7.121818617s) [noise=true , address=145.79.227.37:4567 , piece_id=QHZOH2IPBXQWSBJHBJ4EBJCW5JHHQRURJM4GODY3NP2AOUFWWR5Q, seg_pos={0 0}, upload=469438]: Long tail cancellation due to optimal threshold Dec 27 22:45:07.250951833 (535h16m7.122034879s) [noise=true , address=77.90.31.10:4518 , piece_id=SPJXKTKKCAWKWVJHK34YDRQL7BC4I4GRY4HZYUBL7WZPAT5GYGTA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:58332->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:58332->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58332->77.90.31.10:4518: use of closed network connection Dec 27 22:45:07.250961772 (535h16m7.122044807s) [noise=true , address=77.90.31.10:4518 , piece_id=SPJXKTKKCAWKWVJHK34YDRQL7BC4I4GRY4HZYUBL7WZPAT5GYGTA, seg_pos={0 0}, upload=469438]: Long tail cancellation due to optimal threshold Dec 27 22:45:07.251122168 (535h16m7.122205203s) [noise=true , address=145.79.227.49:4338 , piece_id=UPBP4WKNJB4PG5REJGTAAQEVIRLUVCQHJUV43OTXKT6DGJZGUEHQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 27 22:45:07.251129642 (535h16m7.122212687s) [noise=true , address=145.79.227.49:4338 , piece_id=UPBP4WKNJB4PG5REJGTAAQEVIRLUVCQHJUV43OTXKT6DGJZGUEHQ, seg_pos={0 0}, upload=469438]: Long tail cancellation due to optimal threshold Dec 27 22:45:07.251146795 (535h16m7.12222983s) [seg_pos={0 0}, upload=469438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:45:07.251218642 (535h16m7.122301677s) [upload=469438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:45:07.251223621 (535h16m7.122306666s) [upload=469438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:45:07.559197465 (535h16m7.430280501s) [upload=469438]: Done uploading segments... Dec 27 22:45:07.559217354 (535h16m7.430300389s) [upload=469438]: Upload finished. err: <nil> Duration: 6.358502523s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...................X............X.X...X.....X...X..X....X...] duration: 25.959563503s trace: https://jaeger.dev.storj.io/trace/67e5b65c1836be39 FAIL (2025-12-27 22:47:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: ZSUJHALWAQOHVMTXFW54BB6RF7Z3TVGBUYMAKZIUP4KHVZE2YSTA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 6ZI3SEX5K2F32GL3G6TZ5KZPMWOCLYQGA63JCSIUU334WFQQ3TOQ) context deadline exceeded ======LOGS====== 16m4.620323326s) [upload=469438]: Uploading segments... Dec 27 22:45:04.749488463 (535h16m4.620571508s) [upload=469438]: Got next segment. Inline: false Dec 27 22:45:04.749529381 (535h16m4.620612436s) [upload=469438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:45:04.749536094 (535h16m4.620619149s) [upload=469438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:45:05.000717426 (535h16m4.871800501s) [seg_pos={0 0}, upload=469438]: Begin upload segment... Dec 27 22:45:05.000782259 (535h16m4.871865304s) [seg_pos={0 0}, upload=469438]: Done begin upload segment. Dec 27 22:45:05.000821734 (535h16m4.871904789s) [noise=true , address=77.90.31.10:4518 , piece_id=SPJXKTKKCAWKWVJHK34YDRQL7BC4I4GRY4HZYUBL7WZPAT5GYGTA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.001222127 (535h16m4.872305182s) [noise=true , address=45.91.116.8:4047 , piece_id=OIQP4IXTAED2DLC2X6R6TMBK546W7TXXVFVGRCV3WGDCH65T3FQA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.002353952 (535h16m4.873436997s) [noise=true , address=145.79.227.47:4266 , piece_id=DCQJDVWX7NEWG5R6JHRIUQBEMGZJLNNSBMPKMQPT3L6VOJEWK3EQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.002883460 (535h16m4.873966515s) [noise=true , address=145.79.227.24:4582 , piece_id=ZJQ5GMGRKHLAUV3DZ4LYIUDP7CDJ4C4TTYIB5SK5LGECHEW2ZVSQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.002930680 (535h16m4.874013736s) [noise=true , address=145.79.227.5:4223 , piece_id=WNS3IGWVQAGLWPGEJR7474PEUBVKUWG7ISF3P73TXYE3MNCLECLA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.003303590 (535h16m4.874386645s) [noise=true , address=145.79.227.49:4338 , piece_id=UPBP4WKNJB4PG5REJGTAAQEVIRLUVCQHJUV43OTXKT6DGJZGUEHQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.003723670 (535h16m4.874806715s) [noise=true , address=77.90.31.14:4661 , piece_id=ZSUJHALWAQOHVMTXFW54BB6RF7Z3TVGBUYMAKZIUP4KHVZE2YSTA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.004172565 (535h16m4.87525561s) [noise=true , address=145.79.227.37:4567 , piece_id=QHZOH2IPBXQWSBJHBJ4EBJCW5JHHQRURJM4GODY3NP2AOUFWWR5Q, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.004194817 (535h16m4.875277872s) [noise=true , address=145.79.227.42:4122 , piece_id=ALADJYXJQ4TQTCFKQIN3CHH3GTU72ZB7G3WXX74LIPZN4TD3TD6A, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.004633323 (535h16m4.875716378s) [noise=true , address=145.79.227.9:4365 , piece_id=K2IZ3KP7NM6ZU4BZ6RPEO6LVCTYIM33APM2UWJHPNXGMIWVJ6CXA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.005010059 (535h16m4.876093104s) [noise=true , address=45.91.116.5:4297 , piece_id=6ZI3SEX5K2F32GL3G6TZ5KZPMWOCLYQGA63JCSIUU334WFQQ3TOQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.006339952 (535h16m4.877422997s) [noise=true , address=145.79.227.45:4231 , piece_id=IVFX654533G6O4WXZFAG4ONLB5LIOIVEC6CWE4S3QELFDO5KHDEQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.007234004 (535h16m4.878317049s) [noise=true , address=145.79.227.39:4013 , piece_id=VX4DUSCEWUKC5HTCUVUNQEKOSPR4RJVTPOU7YBSWBPQRTGAZ6RNA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.011044781 (535h16m4.882127826s) [noise=true , address=145.79.227.11:4731 , piece_id=6PL34PORDMFYGTF7KU4ZAHRLO7JU66BFD55FEH23EYKUQRHWTERA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.012865718 (535h16m4.883948763s) [noise=true , address=45.91.116.11:4117 , piece_id=P2HLL4ATCVG2QIOKBJVGO2PFJQCLYA5GLURFGPHPEN4JYRJ5S5LQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.014688018 (535h16m4.885771063s) [noise=true , address=145.79.227.38:4977 , piece_id=ZM7RPMQIVMGTXLO55VGICXWNIEVMUEOZGWSY7HHF3OYUPUSJWEKA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:06.624796567 (535h16m6.495879612s) [upload=469438]: Next returned nil segment Dec 27 22:45:06.624814311 (535h16m6.495897356s) [upload=469438]: Waiting for error group managing segments... Dec 27 22:45:07.009031135 (535h16m6.88011418s) [noise=true , address=145.79.227.11:4731 , piece_id=6PL34PORDMFYGTF7KU4ZAHRLO7JU66BFD55FEH23EYKUQRHWTERA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.012820972 (535h16m6.883904017s) [noise=true , address=145.79.227.24:4582 , piece_id=ZJQ5GMGRKHLAUV3DZ4LYIUDP7CDJ4C4TTYIB5SK5LGECHEW2ZVSQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.019694149 (535h16m6.890777184s) [noise=true , address=145.79.227.39:4013 , piece_id=VX4DUSCEWUKC5HTCUVUNQEKOSPR4RJVTPOU7YBSWBPQRTGAZ6RNA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.020947285 (535h16m6.89203033s) [noise=true , address=145.79.227.9:4365 , piece_id=K2IZ3KP7NM6ZU4BZ6RPEO6LVCTYIM33APM2UWJHPNXGMIWVJ6CXA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.026693517 (535h16m6.897776552s) [noise=true , address=145.79.227.42:4122 , piece_id=ALADJYXJQ4TQTCFKQIN3CHH3GTU72ZB7G3WXX74LIPZN4TD3TD6A, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.027854889 (535h16m6.898937924s) [noise=true , address=145.79.227.5:4223 , piece_id=WNS3IGWVQAGLWPGEJR7474PEUBVKUWG7ISF3P73TXYE3MNCLECLA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.028395879 (535h16m6.899478924s) [noise=true , address=145.79.227.38:4977 , piece_id=ZM7RPMQIVMGTXLO55VGICXWNIEVMUEOZGWSY7HHF3OYUPUSJWEKA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.032625914 (535h16m6.903708949s) [noise=true , address=145.79.227.45:4231 , piece_id=IVFX654533G6O4WXZFAG4ONLB5LIOIVEC6CWE4S3QELFDO5KHDEQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.048007063 (535h16m6.919090098s) [noise=true , address=145.79.227.47:4266 , piece_id=DCQJDVWX7NEWG5R6JHRIUQBEMGZJLNNSBMPKMQPT3L6VOJEWK3EQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.167310827 (535h16m7.038393872s) [noise=true , address=77.90.31.14:4661 , piece_id=ZSUJHALWAQOHVMTXFW54BB6RF7Z3TVGBUYMAKZIUP4KHVZE2YSTA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.221778913 (535h16m7.092861948s) [noise=true , address=45.91.116.5:4297 , piece_id=6ZI3SEX5K2F32GL3G6TZ5KZPMWOCLYQGA63JCSIUU334WFQQ3TOQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.244653280 (535h16m7.115736325s) [noise=true , address=45.91.116.11:4117 , piece_id=P2HLL4ATCVG2QIOKBJVGO2PFJQCLYA5GLURFGPHPEN4JYRJ5S5LQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.250513118 (535h16m7.121596153s) [noise=true , address=45.91.116.8:4047 , piece_id=OIQP4IXTAED2DLC2X6R6TMBK546W7TXXVFVGRCV3WGDCH65T3FQA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.250536132 (535h16m7.121619177s) [seg_pos={0 0}, upload=469438]: Segment reached optimal threshold of 13 pieces. Dec 27 22:45:07.250729120 (535h16m7.121812165s) [noise=true , address=145.79.227.37:4567 , piece_id=QHZOH2IPBXQWSBJHBJ4EBJCW5JHHQRURJM4GODY3NP2AOUFWWR5Q, seg_pos={0 0}, upload=469438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:54026->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:54026->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54026->145.79.227.37:4567: use of closed network connection Dec 27 22:45:07.250735572 (535h16m7.121818617s) [noise=true , address=145.79.227.37:4567 , piece_id=QHZOH2IPBXQWSBJHBJ4EBJCW5JHHQRURJM4GODY3NP2AOUFWWR5Q, seg_pos={0 0}, upload=469438]: Long tail cancellation due to optimal threshold Dec 27 22:45:07.250951833 (535h16m7.122034879s) [noise=true , address=77.90.31.10:4518 , piece_id=SPJXKTKKCAWKWVJHK34YDRQL7BC4I4GRY4HZYUBL7WZPAT5GYGTA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:58332->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:58332->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58332->77.90.31.10:4518: use of closed network connection Dec 27 22:45:07.250961772 (535h16m7.122044807s) [noise=true , address=77.90.31.10:4518 , piece_id=SPJXKTKKCAWKWVJHK34YDRQL7BC4I4GRY4HZYUBL7WZPAT5GYGTA, seg_pos={0 0}, upload=469438]: Long tail cancellation due to optimal threshold Dec 27 22:45:07.251122168 (535h16m7.122205203s) [noise=true , address=145.79.227.49:4338 , piece_id=UPBP4WKNJB4PG5REJGTAAQEVIRLUVCQHJUV43OTXKT6DGJZGUEHQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 27 22:45:07.251129642 (535h16m7.122212687s) [noise=true , address=145.79.227.49:4338 , piece_id=UPBP4WKNJB4PG5REJGTAAQEVIRLUVCQHJUV43OTXKT6DGJZGUEHQ, seg_pos={0 0}, upload=469438]: Long tail cancellation due to optimal threshold Dec 27 22:45:07.251146795 (535h16m7.12222983s) [seg_pos={0 0}, upload=469438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:45:07.251218642 (535h16m7.122301677s) [upload=469438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:45:07.251223621 (535h16m7.122306666s) [upload=469438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:45:07.559197465 (535h16m7.430280501s) [upload=469438]: Done uploading segments... Dec 27 22:45:07.559217354 (535h16m7.430300389s) [upload=469438]: Upload finished. err: <nil> Duration: 25.959588891s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 208.315128ms trace: https://jaeger.dev.storj.io/trace/65d071dc3d109c6 FAIL (2025-12-27 22:46:42 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 208.342079ms core uplink ap1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)